Re: Floating point calculus error...

["Bart Schaefer" <schaefer@brasslantern.com>]

On Aug 25,  5:22pm, Patrick Aussems wrote:
}
} I was doing some shell scripts that were supposed to add CPU usages from
} the ps output... But I got puzzled when I got an answer with a 17th
} decimal while adding numbers with only 1 decimal... Strange isn't it?

Not really.

schaefer[507] print -l $[0.1] $[0.2] $[0.3] $[0.4]
0.10000000000000001
0.20000000000000001
0.29999999999999999
0.40000000000000002
schaefer[508] typeset -F 4 f
schaefer[509] for f in $[0.1] $[0.2] $[0.3] $[0.4]; print $f
0.1000
0.2000
0.3000
0.4000

} The value returned is: 0.80000000000000004 which is obviously not the
} correct answer... Okay, floating points numbers are inaccurate, but
} still...

Yes, floating point numbers are inaccurate.  Zsh simply doesn't round
when inside arithmetic expressions, only when outputting float-formatted
parameters where it knows how many decimal places to use (default 10, but
I forced 4 above).

For comparison, see what happens when the number is computed by floating
point addition rather than by explicit conversion from decimal.  In the
first example below (history number [510]) I've used $(($f)) to force zsh
to round down to 1 decimal place and then convert back to float again.
In the second case, $((f)) evaluates f entirely in math context, using as
much precision as possible, which also shows why the first case didn't stop
at 0.9.

schaefer[510] typeset -F 1 f ; for ((f=0; f < 1; f += 0.1)) print $f = $(($f))
0.0 = 0
0.1 = 0.10000000000000001
0.2 = 0.20000000000000001
0.3 = 0.29999999999999999
0.4 = 0.40000000000000002
0.5 = 0.5
0.6 = 0.59999999999999998
0.7 = 0.69999999999999996
0.8 = 0.80000000000000004
0.9 = 0.90000000000000002
1.0 = 1
schaefer[511] typeset -F 1 f ; for ((f=0; f < 1; f += 0.1)) print $f = $((f)) 
0.0 = 0
0.1 = 0.10000000000000001
0.2 = 0.20000000000000001
0.3 = 0.30000000000000004
0.4 = 0.40000000000000002
0.5 = 0.5
0.6 = 0.59999999999999998
0.7 = 0.69999999999999996
0.8 = 0.79999999999999993
0.9 = 0.89999999999999991
1.0 = 0.99999999999999989

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net