floating-point precision and zsh vs ksh93 and coreutils

floating-point precision and zsh vs ksh93 and coreutils

[Vincent Lefevre]

For floating-point arithmetic evaluation, it appears that zsh uses the
"double" type while ksh93 and GNU coreutils both use the "long double"
type, which means incompatibility:

$ ksh93 -c '/usr/bin/printf "%a\n" $((1./3))'
0xa.aaaaaaaaaaaaa9ep-5
$ zsh -fc '/usr/bin/printf "%a\n" $((1./3))'
0xa.aaaaaaaaaaaa74ep-5

and

$ ksh93 -c '/usr/bin/printf "%a\n" $((43./2**22))'
0xa.cp-20
$ zsh -fc '/usr/bin/printf "%a\n" $((43./2**22))'
0xa.c0000000000025cp-20

In the second case, while the number is exactly representable
(and has the same value) in both double and long double, the
issue is visible because the number is passed with a decimal
representation, with enough decimal digits to read the result
with 53-bit precision in the zsh test.

Shouldn't zsh switch to long double?

Note: using printf from the GNU coreutils instead of the zsh printf
builtin is needed for %a support.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: floating-point precision and zsh vs ksh93 and coreutils

[Stephane Chazelas]

2022-03-23 17:36:46 +0100, Vincent Lefevre:
[...]
> Shouldn't zsh switch to long double?
[...]

I'm not convinced it's a good idea. I had made a long write-up
related to that some time ago at
https://unix.stackexchange.com/questions/422122/why-does-0-1-expand-to-0-10000000000000001-in-zsh
that we did discussed it here about the artefacts whereby echo
$((0.1)) outputs 0.10000000000000001 for instance.

Several problems:
- double is still the most commonly used float representation
  used by default in other languages (perl, python, most if not
  all awks...)
- long double precision varies with the system/compiler/CPU
  (64bit, 80bit, 128bit), whilst double is more consistently
  64bit on Unix systems at least.
- so to preserve the precision upon round-trip to/from decimal,
  we'd need 17, 21 or 36 digit precision making for even uglier
  artefacts (and very long numbers), and if we give up on
  preserving precision, we get the same problems as affect
  yash/ksh93 described at that link above.

  $ ksh -c 'if (( $((1. / 3)) == 1./3 )); then echo yes; else echo no; fi'
  no
  $ ksh -c 'echo $(( $((1. / 3)) - 1./3 ))'
  -3.52365706057788941e-19

  See the discussion around workers/45106 (to which you
  contributed) about how to reduce the display artefacts, but
  that's potentially costly considering that shells do have to
  always go back and forth between binary and decimal
  representation of numbers, as decimal is the "interchange"
  format to pass to the user or to commands and that's the
  decimal text representation that is stored in variable (IIRC
  ksh93 stores both the decimal/text and binary representation
  though for variables declared with typeset -F/E to reduce the
  need to translate all the time).

To me, if double is good enough for perl or python, it should we
all the more for a shell.

-- 
Stephane

Re: floating-point precision and zsh vs ksh93 and coreutils

[Bart Schaefer]

On Sun, Mar 27, 2022 at 1:27 AM Stephane Chazelas <stephane@chazelas.org> wrote:
>
>   See the discussion around workers/45106 (to which you
>   contributed) about how to reduce the display artefacts

Vincent's final contribution to that discussion was a link to the
dtoa() implementation in C by D.Gay ... which just for comparison is
more than 6000 lines of code or about 4% the size of the entire zsh C
code at the time I write this.

Re: floating-point precision and zsh vs ksh93 and coreutils

[Vincent Lefevre]

On 2022-03-27 09:26:52 +0100, Stephane Chazelas wrote:
> 2022-03-23 17:36:46 +0100, Vincent Lefevre:
> [...]
> > Shouldn't zsh switch to long double?
> [...]
> 
> I'm not convinced it's a good idea. I had made a long write-up
> related to that some time ago at
> https://unix.stackexchange.com/questions/422122/why-does-0-1-expand-to-0-10000000000000001-in-zsh
> that we did discussed it here about the artefacts whereby echo
> $((0.1)) outputs 0.10000000000000001 for instance.
> 
> Several problems:
> - double is still the most commonly used float representation
>   used by default in other languages (perl, python, most if not
>   all awks...)
> - long double precision varies with the system/compiler/CPU
>   (64bit, 80bit, 128bit), whilst double is more consistently
>   64bit on Unix systems at least.
> - so to preserve the precision upon round-trip to/from decimal,
>   we'd need 17, 21 or 36 digit precision making for even uglier
>   artefacts (and very long numbers), and if we give up on
>   preserving precision, we get the same problems as affect
>   yash/ksh93 described at that link above.
> 
>   $ ksh -c 'if (( $((1. / 3)) == 1./3 )); then echo yes; else echo no; fi'
>   no
>   $ ksh -c 'echo $(( $((1. / 3)) - 1./3 ))'
>   -3.52365706057788941e-19

OK, I forgot that ksh was actually buggy. I thought that it output the
minimum number of digits that preserves the round trip back to binary.

Now, given that ksh doesn't even use a "long double compatible"
format for its output, I don't see any reason why coreutils uses
long double by default (IMHO, it should use a length modifier
for that, as in C).

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: floating-point precision and zsh vs ksh93 and coreutils

[Vincent Lefevre]

On 2022-03-27 15:26:42 -0700, Bart Schaefer wrote:
> Vincent's final contribution to that discussion was a link to the
> dtoa() implementation in C by D.Gay ... which just for comparison is
> more than 6000 lines of code or about 4% the size of the entire zsh C
> code at the time I write this.

I haven't looked at the code recently, but I thought that D. Gay's
code provided several functions. For just the output to decimal,
I suppose that this should be much shorter.

BTW, it would be even better if such a function were standardized
and in the standard library.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: floating-point precision and zsh vs ksh93 and coreutils

[Bart Schaefer]

On Thu, Apr 7, 2022 at 8:07 AM Vincent Lefevre <vincent@vinc17.net> wrote:
>
> On 2022-03-27 15:26:42 -0700, Bart Schaefer wrote:
> > Vincent's final contribution to that discussion was a link to the
> > dtoa() implementation in C by D.Gay ... which just for comparison is
> > more than 6000 lines of code or about 4% the size of the entire zsh C
> > code at the time I write this.
>
> I haven't looked at the code recently, but I thought that D. Gay's
> code provided several functions. For just the output to decimal,
> I suppose that this should be much shorter.

Actually, no.  I visited the site; each of the "several functions" is
isolated to its own file.  6k lines is JUST dtoa().  About a third of
it is huge lookup tables.