Floating-point arithmetic evaluation on Linux/x86 (traditional FPU)

Floating-point arithmetic evaluation on Linux/x86 (traditional FPU)

[Vincent Lefevre]

The zshmisc man page says:

ARITHMETIC EVALUATION
  The  shell  can  perform  integer and floating point arithmetic, either
  using the builtin let, or via a substitution of the form $((...)).  For
  integers,  the  shell is usually compiled to use 8-byte precision where
  this is available, otherwise precision is 4 bytes.  This can be tested,
  for example, by giving the command `print - $(( 12345678901 ))'; if the
  number appears unchanged, the precision is at least 8 bytes.   Floating
  point arithmetic is always double precision.

But it is not true that zsh always uses double precision. It seems to
use the C double type, but this doesn't correspond to double precision
by default under Linux/x86 (when the traditional FPU is used, i.e. no
SSE2). For instance, I get:

vin:~> uname -a
Linux vin 2.6.22-1-686-bigmem #1 SMP Sun Jul 29 15:54:01 UTC 2007 i686 GNU/Linux
vin:~> echo $((9007199254740994.0+(1.0-1/65536.0)-9007199254740994.0))
2.

instead of "0." on other machines, due to the "double rounding"
effect. More information about this problem on:

  http://www.vinc17.org/research/extended.en.html

Either zsh should switch to double precision (assuming that modules
and so on won't change the precision back to the extended precision),
or the zsh documentation should be changed.

-- 
Vincent Lefèvre <vincent@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / Arenaire project (LIP, ENS-Lyon)

Re: Floating-point arithmetic evaluation on Linux/x86 (traditional FPU)

[Peter Stephenson]

Vincent Lefevre wrote:
> But it is not true that zsh always uses double precision. It seems to
> use the C double type, but this doesn't correspond to double precision
> by default under Linux/x86 (when the traditional FPU is used, i.e. no
> SSE2).
> 
> Either zsh should switch to double precision (assuming that modules
> and so on won't change the precision back to the extended precision),
> or the zsh documentation should be changed.

There's no hope of me taking on another architectural investigation, so
unless someone can look at this, then as long as it's simply a question
of restating what the shell does it's easy.

Index: Doc/Zsh/arith.yo
===================================================================
RCS file: /cvsroot/zsh/zsh/Doc/Zsh/arith.yo,v
retrieving revision 1.11
diff -u -r1.11 arith.yo
--- Doc/Zsh/arith.yo	15 Jun 2007 10:02:24 -0000	1.11
+++ Doc/Zsh/arith.yo	12 Nov 2007 16:40:38 -0000
@@ -12,7 +12,8 @@
 is available, otherwise precision is 4 bytes.  This can be tested, for
 example, by giving the command `tt(print - $(( 12345678901 )))'; if the
 number appears unchanged, the precision is at least 8 bytes.  Floating
-point arithmetic is always double precision.
+point arithmetic always uses the `double' type with whatever corresponding
+precision is provided by the compiler and the library.
 
 The tt(let) builtin command takes arithmetic expressions as arguments; each
 is evaluated separately.  Since many of the arithmetic operators, as well

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070