LC_NUMERIC=fr_FR and floating point arithmetics

LC_NUMERIC=fr_FR and floating point arithmetics

[Stephane CHAZELAS]

$ LC_NUMERIC=fr_FR ksh93 -c 'float a=1; echo $(( a / 3 ))'
0,333333333333
$ LC_NUMERIC=fr_FR zsh -c 'float a=1; echo $(( a / 3 ))'
0,33333333333333331.

zsh seems to assume that "." is the decimal separator which is
not correct in a french locale.

I agree this is confusing. A ksh93 script such as
echo $(( 1. / 3 ))

won't work under french locale.
(must be echo $(( 1, / 3 )) )

(tried with zsh 4.1.0-dev-7)

-- 
Stéphane

Re: LC_NUMERIC=fr_FR and floating point arithmetics

[Oliver Kiddle]

On 24 Feb, Stephane CHAZELAS wrote:
> $ LC_NUMERIC=fr_FR ksh93 -c 'float a=1; echo $(( a / 3 ))'
> 0,333333333333
> $ LC_NUMERIC=fr_FR zsh -c 'float a=1; echo $(( a / 3 ))'
> 0,33333333333333331.
>
> zsh seems to assume that "." is the decimal separator which is
> not correct in a french locale.
> 
> I agree this is confusing. A ksh93 script such as
> echo $(( 1. / 3 ))
>
> won't work under french locale.
> (must be echo $(( 1, / 3 )) )

Which is worse in my opinion. $(( 1.1 )) really ought to be interpreted
as a 1 point 1 regardless of locale - it is otherwise impossible to use
decimals in portable shell scripts.

If we didn't have backward and ksh compatibility to consider, I would
be inclined to say that $a for floats and $(( ... )) should always use
the C locale and anyone wanting locale specific output could use printf
(see below) (implementing this is easy; just save the LC_NUMERIC locale
in convfloat()).

But keeping locale handling in output means that scalar parameters
can't be reused in a calculation. And making math evaluation accept
either `.' or whatever the locale dictates is not easy when you
consider that a comma in math evaluation is used as a separator:
  (( a=1, 3.4 ))
Also, because we allow this:
  % a='3 + 4'
  % echo $(( a ))
  7
you can't just interpret the comma in a parameter expansion. So I can't
think of any solution except limiting math evaluation to the C locale.

So what can we do?

I did a grep for mon_decimal_point in /usr/share/i18n/locales to see
if any other characters are used as decimal separators. It indicates
that Portugal use `$' as their decimal separator. Seems weird. Apart
from that it is , or . for the rest of the world except Burkina Faso
who apparently use something outside the normal ASCII range.

Currently, a bug prevents locale handling for printf (on at least some
platforms). The bug causing this is in math.c where it does:
	prev_locale = setlocale(LC_NUMERIC, NULL);
	setlocale(LC_NUMERIC, "POSIX");
The second setlocale call clobbers prev_locale - we have to use
dupstring(). The patch fixes this.

Oliver

diff -ur zsh-latest/Src/math.c localeprob2/Src/math.c
--- zsh-latest/Src/math.c	2002-12-19 10:58:19.000000000 +0000
+++ localeprob2/Src/math.c	2003-03-09 01:13:18.000000000 +0000
@@ -399,12 +399,12 @@
 		    /* it's a float */
 		    yyval.type = MN_FLOAT;
 #ifdef USE_LOCALE
-		    prev_locale = setlocale(LC_NUMERIC, NULL);
+		    prev_locale = dupstring(setlocale(LC_NUMERIC, NULL));
 		    setlocale(LC_NUMERIC, "POSIX");
 #endif
 		    yyval.u.d = strtod(ptr, &nptr);
 #ifdef USE_LOCALE
-		    setlocale(LC_NUMERIC, prev_locale);
+		    if (prev_locale) setlocale(LC_NUMERIC, prev_locale);
 #endif
 		    if (ptr == nptr || *nptr == '.') {
 			zerr("bad floating point constant", NULL, 0);

Re: LC_NUMERIC=fr_FR and floating point arithmetics

[Zefram]

Oliver Kiddle wrote:
>But keeping locale handling in output means that scalar parameters
>can't be reused in a calculation.

Numeric output should by default be in the input format (C locale)
so that it can be reused in the expected manner.  If printf is not
sufficient for rendering numbers in human format, we could add an
output-using-locale flag to the output format specification syntax of
$(()) -- there's already syntax to select the radix to use on output.

-zefram

Re: LC_NUMERIC=fr_FR and floating point arithmetics

[Oliver Kiddle]

Zefram wrote:
> 
> Numeric output should by default be in the input format (C locale)
> so that it can be reused in the expected manner.

Okay, this patch makes it do that.

> If printf is not
> sufficient for rendering numbers in human format, we could add an
> output-using-locale flag to the output format specification syntax of
> $(()) -- there's already syntax to select the radix to use on output.

It would possibly be more useful for $a than $(()) because that is where
all the typeset -F/-E stuff applies. I'll wait and see if there is
demand and if so, a parameter expansion flag could also be added (L and
l are gone along with most of the rest of the alphabet so we'd need to
think of a suitable free letter). If ksh keeps it's current behaviour, a
KSH_something option could perhaps be used.

I consider this patch a bug fix which would imply that I should apply it
to 4.0 but 4.0 doesn't have printf and hence has no way to output floats
in the current locale. I could backport printf, leave the patch out of
4.0 or do something else. Any thoughts?

Note also, that there is no way to input numbers with locale
conventions. If there is demand for a way to do this, we could perhaps
add an option to read so that typeset -F num; read num would do this but
lose math evaluation. Or perhaps add the facility somewhere else.

Oliver

Index: Src/params.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/params.c,v
retrieving revision 1.69
diff -u -r1.69 params.c
--- Src/params.c	31 Oct 2002 18:32:40 -0000	1.69
+++ Src/params.c	11 Mar 2003 10:00:10 -0000
@@ -3417,6 +3417,7 @@
 convfloat(double dval, int digits, int flags, FILE *fout)
 {
     char fmt[] = "%.*e";
+    char *prev_locale, *ret;
 
     /*
      * The difficulty with the buffer size is that a %f conversion
@@ -3451,16 +3452,24 @@
 	    digits--;
 	}
     }
+#ifdef USE_LOCALE
+    prev_locale = dupstring(setlocale(LC_NUMERIC, NULL));
+    setlocale(LC_NUMERIC, "POSIX");
+#endif
     if (fout) {
 	fprintf(fout, fmt, digits, dval);
-	return NULL;
+	ret = NULL;
     } else {
 	VARARR(char, buf, 512 + digits);
 	sprintf(buf, fmt, digits, dval);
 	if (!strchr(buf, 'e') && !strchr(buf, '.'))
 	    strcat(buf, ".");
-	return dupstring(buf);
+	ret = dupstring(buf);
     }
+#ifdef USE_LOCALE
+    if (prev_locale) setlocale(LC_NUMERIC, prev_locale);
+#endif
+    return ret;
 }
 
 /* Start a parameter scope */

Re: LC_NUMERIC=fr_FR and floating point arithmetics

[Philippe Troin]

Oliver Kiddle <okiddle@yahoo.co.uk> writes:

> Index: Src/params.c
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Src/params.c,v
> retrieving revision 1.69
> diff -u -r1.69 params.c
> --- Src/params.c	31 Oct 2002 18:32:40 -0000	1.69
> +++ Src/params.c	11 Mar 2003 10:00:10 -0000
> @@ -3417,6 +3417,7 @@
>  convfloat(double dval, int digits, int flags, FILE *fout)
>  {
>      char fmt[] = "%.*e";
> +    char *prev_locale, *ret;
>  
>      /*
>       * The difficulty with the buffer size is that a %f conversion
> @@ -3451,16 +3452,24 @@
>  	    digits--;
>  	}
>      }
> +#ifdef USE_LOCALE
> +    prev_locale = dupstring(setlocale(LC_NUMERIC, NULL));
> +    setlocale(LC_NUMERIC, "POSIX");
> +#endif
>      if (fout) {
>  	fprintf(fout, fmt, digits, dval);
> -	return NULL;
> +	ret = NULL;
>      } else {
>  	VARARR(char, buf, 512 + digits);
>  	sprintf(buf, fmt, digits, dval);
>  	if (!strchr(buf, 'e') && !strchr(buf, '.'))
>  	    strcat(buf, ".");
> -	return dupstring(buf);
> +	ret = dupstring(buf);
>      }
> +#ifdef USE_LOCALE
> +    if (prev_locale) setlocale(LC_NUMERIC, prev_locale);
> +#endif
> +    return ret;
>  }
>  
>  /* Start a parameter scope */

Aren't you leaking a copy of the current locale (via dupstring())
every time?

Phil.

Re: LC_NUMERIC=fr_FR and floating point arithmetics

[Oliver Kiddle]

Philippe Troin wrote:
> 
> Aren't you leaking a copy of the current locale (via dupstring())
> every time?

dupstring() calls zhalloc() to allocate memory instead of zalloc()
(unlike ztrdup()). So the memory should be freed the next time popheap()
is called (which I think is done for each command line). The comment at
the top of mem.c is what I'm basing this on. So I've always assumed that
zalloc should only be used for things which need to persist from one
command to the next, where realloc() is needed or where a large chunk of
memory is needed.

Oliver