From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <zsh-users-return-5289-mason-zsh=primenet.com.au@sunsite.dk>
Received: (qmail 11008 invoked from network); 25 Aug 2002 16:38:17 -0000
Received: from sunsite.dk (130.225.247.90)
  by ns1.primenet.com.au with SMTP; 25 Aug 2002 16:38:17 -0000
Received: (qmail 19656 invoked by alias); 25 Aug 2002 16:38:08 -0000
Mailing-List: contact zsh-users-help@sunsite.dk; run by ezmlm
Precedence: bulk
X-No-Archive: yes
X-Seq: 5289
Received: (qmail 19642 invoked from network); 25 Aug 2002 16:38:07 -0000
From: "Bart Schaefer" <schaefer@brasslantern.com>
Message-Id: <1020825163707.ZM32042@candle.brasslantern.com>
Date: Sun, 25 Aug 2002 16:37:06 +0000
In-Reply-To: <1030288941.517.8.camel@Amok>
Comments: In reply to Patrick Aussems <zyk@bugfactory.org>
        "Floating point calculus error..." (Aug 25,  5:22pm)
References: <1030288941.517.8.camel@Amok>
X-Mailer: Z-Mail (5.0.0 30July97)
To: Patrick Aussems <zyk@bugfactory.org>, zsh-users@sunsite.dk
Subject: Re: Floating point calculus error...
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

On Aug 25,  5:22pm, Patrick Aussems wrote:
}
} I was doing some shell scripts that were supposed to add CPU usages from
} the ps output... But I got puzzled when I got an answer with a 17th
} decimal while adding numbers with only 1 decimal... Strange isn't it?

Not really.

schaefer[507] print -l $[0.1] $[0.2] $[0.3] $[0.4]
0.10000000000000001
0.20000000000000001
0.29999999999999999
0.40000000000000002
schaefer[508] typeset -F 4 f
schaefer[509] for f in $[0.1] $[0.2] $[0.3] $[0.4]; print $f
0.1000
0.2000
0.3000
0.4000

} The value returned is: 0.80000000000000004 which is obviously not the
} correct answer... Okay, floating points numbers are inaccurate, but
} still...

Yes, floating point numbers are inaccurate.  Zsh simply doesn't round
when inside arithmetic expressions, only when outputting float-formatted
parameters where it knows how many decimal places to use (default 10, but
I forced 4 above).

For comparison, see what happens when the number is computed by floating
point addition rather than by explicit conversion from decimal.  In the
first example below (history number [510]) I've used $(($f)) to force zsh
to round down to 1 decimal place and then convert back to float again.
In the second case, $((f)) evaluates f entirely in math context, using as
much precision as possible, which also shows why the first case didn't stop
at 0.9.

schaefer[510] typeset -F 1 f ; for ((f=0; f < 1; f += 0.1)) print $f = $(($f))
0.0 = 0
0.1 = 0.10000000000000001
0.2 = 0.20000000000000001
0.3 = 0.29999999999999999
0.4 = 0.40000000000000002
0.5 = 0.5
0.6 = 0.59999999999999998
0.7 = 0.69999999999999996
0.8 = 0.80000000000000004
0.9 = 0.90000000000000002
1.0 = 1
schaefer[511] typeset -F 1 f ; for ((f=0; f < 1; f += 0.1)) print $f = $((f)) 
0.0 = 0
0.1 = 0.10000000000000001
0.2 = 0.20000000000000001
0.3 = 0.30000000000000004
0.4 = 0.40000000000000002
0.5 = 0.5
0.6 = 0.59999999999999998
0.7 = 0.69999999999999996
0.8 = 0.79999999999999993
0.9 = 0.89999999999999991
1.0 = 0.99999999999999989

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net