Re: zsh converts a floating-point number to string with too much precision

[Roman Perepelitsa <roman.perepelitsa@gmail.com>]

On Sat, Dec 21, 2019 at 9:48 AM Stephane Chazelas
<stephane.chazelas@gmail.com> wrote:
> So on a system (with a compiler) where C doubles are implemented
> as IEEE 754 double precision, both 1.1 and 1.1000000000000001
> are represented as the same binary double (whose exact value is
> 1.100000000000000088817841970012523233890533447265625).
>
> So you're saying echo $((1.1000000000000001)) and echo $((1.1))
> should output 1.1, because even though 1.1000000000000001 is
> closer to that value than 1.1000000000000000, zsh should pick
> the latter because people prefer to see shorter number
> representations and in that case it doesn't matter which one we
> pick as both lead to the same double.

Correct. The best formal description of this process that I know of is
the specification of to_chars() functions in the C++ standard:

    The functions [...] ensure that the string representation consists
    of the smallest number of characters such that there is at least
    one digit before the radix point (if present) and parsing the
    representation using the corresponding from_chars function
    recovers value exactly. [Note: This guarantee applies only if
    to_chars and from_chars are executed on the same implementation.
    — end note] If there are several such representations, the
    representation with the smallest difference from the
    floating-point argument value is chosen, resolving any remaining
    ties using rounding according to round_to_nearest.

There is no simple algorithm that achieves this. I recall reading long
papers a few years back that were describing various inventions in
this field. They were pretty scary.

> Is there a standard C API for that?

Not that I know of. FWIW, neither gcc (libstdc++) nor clang (libc++)
have implemented to_chars() for floating points in the two years since
it's been added to the standard (C++17).

>
> Or would we get the output of sprintf("%.17g"), look at the last
> two significant digits, if the second last is 9 or 0, then see
> if rounding it and doing a strtod again yields the same double?

I believe printing the value with full precision and then truncating
the string this is the most popular approach. All implementations that
are faster are only marginally so, and they are much more complex. I
think this approach should work well enough for zsh. Lifting this sort
of implementation from another project would be ideal (someone would
have to find it; my apologies for not doing it myself).

Roman.