From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 10585 invoked from network); 25 Aug 2002 15:54:32 -0000 Received: from sunsite.dk (130.225.247.90) by ns1.primenet.com.au with SMTP; 25 Aug 2002 15:54:32 -0000 Received: (qmail 5919 invoked by alias); 25 Aug 2002 15:54:21 -0000 Mailing-List: contact zsh-users-help@sunsite.dk; run by ezmlm Precedence: bulk X-No-Archive: yes X-Seq: 5285 Received: (qmail 5905 invoked from network); 25 Aug 2002 15:54:20 -0000 X-Amavix: Anti-spam check done by SpamAssassin X-Amavix: Anti-virus check done by McAfee X-Amavix: Scanned by Amavix X-Authentication-Warning: greux.loria.fr: lefevre set sender to vincent@vinc17.org using -f Date: Sun, 25 Aug 2002 17:53:50 +0200 From: Vincent Lefevre To: Patrick Aussems Cc: zsh-users@sunsite.auc.dk Subject: Re: Floating point calculus error... Message-ID: <20020825155349.GD6783@greux.loria.fr> References: <1030288941.517.8.camel@Amok> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1030288941.517.8.camel@Amok> X-Mailer-Info: http://www.vinc17.org/mutt/ User-Agent: Mutt/1.5.1-current-20020801i On Sun, Aug 25, 2002 at 17:22:20 +0200, Patrick Aussems wrote: > I was doing some shell scripts that were supposed to add CPU usages from > the ps output... But I got puzzled when I got an answer with a 17th > decimal while adding numbers with only 1 decimal... Strange isn't it? The numbers are internally represented in binary and 0.4 isn't exactly representable in binary... But here, this is a problem with zsh, which gives an additional decimal digit. You can see with: $ echo $[0.4] 0.40000000000000002 I think that it should give the minimum number of decimals in such a way that when converting back to binary, one should get the same result. -- Vincent Lefèvre - Web: - 100% validated (X)HTML - Acorn Risc PC, Yellow Pig 17, Championnat International des Jeux Mathématiques et Logiques, TETRHEX, etc. Work: CR INRIA - computer arithmetic / SPACES project at LORIA