Re: floating-point precision and zsh vs ksh93 and coreutils

[Stephane Chazelas <stephane@chazelas.org>]

2022-03-23 17:36:46 +0100, Vincent Lefevre:
[...]
> Shouldn't zsh switch to long double?
[...]

I'm not convinced it's a good idea. I had made a long write-up
related to that some time ago at
https://unix.stackexchange.com/questions/422122/why-does-0-1-expand-to-0-10000000000000001-in-zsh
that we did discussed it here about the artefacts whereby echo
$((0.1)) outputs 0.10000000000000001 for instance.

Several problems:
- double is still the most commonly used float representation
  used by default in other languages (perl, python, most if not
  all awks...)
- long double precision varies with the system/compiler/CPU
  (64bit, 80bit, 128bit), whilst double is more consistently
  64bit on Unix systems at least.
- so to preserve the precision upon round-trip to/from decimal,
  we'd need 17, 21 or 36 digit precision making for even uglier
  artefacts (and very long numbers), and if we give up on
  preserving precision, we get the same problems as affect
  yash/ksh93 described at that link above.

  $ ksh -c 'if (( $((1. / 3)) == 1./3 )); then echo yes; else echo no; fi'
  no
  $ ksh -c 'echo $(( $((1. / 3)) - 1./3 ))'
  -3.52365706057788941e-19

  See the discussion around workers/45106 (to which you
  contributed) about how to reduce the display artefacts, but
  that's potentially costly considering that shells do have to
  always go back and forth between binary and decimal
  representation of numbers, as decimal is the "interchange"
  format to pass to the user or to commands and that's the
  decimal text representation that is stored in variable (IIRC
  ksh93 stores both the decimal/text and binary representation
  though for variables declared with typeset -F/E to reduce the
  need to translate all the time).

To me, if double is good enough for perl or python, it should we
all the more for a shell.

-- 
Stephane