From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <zsh-users-return-5287-mason-zsh=primenet.com.au@sunsite.dk>
Received: (qmail 10968 invoked from network); 25 Aug 2002 16:32:53 -0000
Received: from sunsite.dk (130.225.247.90)
  by ns1.primenet.com.au with SMTP; 25 Aug 2002 16:32:53 -0000
Received: (qmail 17369 invoked by alias); 25 Aug 2002 16:32:39 -0000
Mailing-List: contact zsh-users-help@sunsite.dk; run by ezmlm
Precedence: bulk
X-No-Archive: yes
X-Seq: 5287
Received: (qmail 17355 invoked from network); 25 Aug 2002 16:32:39 -0000
Cc: zsh-users@sunsite.auc.dk
To: Patrick Aussems <zyk@bugfactory.org>,
	Vincent Lefevre <vincent@vinc17.org>
Subject: Re: Floating point calculus error...
References: <1030288941.517.8.camel@Amok>
	<20020825155349.GD6783@greux.loria.fr>
From: David =?iso-8859-1?q?K=E5gedal?= <davidk@lysator.liu.se>
Date: Sun, 25 Aug 2002 18:32:38 +0200
In-Reply-To: <20020825155349.GD6783@greux.loria.fr> (Vincent Lefevre's
 message of "Sun, 25 Aug 2002 17:53:50 +0200")
Message-ID: <87k7me2921.fsf@bix.grotte>
User-Agent: Gnus/5.090007 (Oort Gnus v0.07) Emacs/21.2
 (i386-debian-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Vincent Lefevre <vincent@vinc17.org> writes:

> On Sun, Aug 25, 2002 at 17:22:20 +0200, Patrick Aussems wrote:
>> I was doing some shell scripts that were supposed to add CPU usages from
>> the ps output... But I got puzzled when I got an answer with a 17th
>> decimal while adding numbers with only 1 decimal... Strange isn't it?
>
> The numbers are internally represented in binary and 0.4 isn't exactly
> representable in binary... But here, this is a problem with zsh, which
> gives an additional decimal digit. You can see with:
>
> $ echo $[0.4]
> 0.40000000000000002
>
> I think that it should give the minimum number of decimals in such a way
> that when converting back to binary, one should get the same result.

But it does.

$ echo $[.40000000000000002]
0.40000000000000002

The point is, as you say, that 0.4 can't be represented as an exact
binary floating-point number, no matter how large the representation
is.  Just like 1/3 can't be represented as a decimal floating-point
number.

--=20
David K=E5gedal