Re: Possible bug in setlocale upon invalid LC_ALL value

[Assaf Gordon <assafgordon@gmail.com>]

Hello Rich,

thank you for the prompt and detailed response.

> On Apr 1, 2016, at 20:58, Rich Felker <dalias@libc.org> wrote:
> 
> On Fri, Apr 01, 2016 at 08:47:01PM -0400, Assaf Gordon wrote:
>> I think I've encountered a problem in musl, where using setlocale with invalid locale name returns the invalid locale instead of a known locale.
> 
> This is intentional. All locale names are valid under musl, and those
> which don't have any particular definition are just aliases for
> C.UTF-8.

I will suggest a minor fix to GNU coreutils to accommodate for this current implementation.

> The alternative would be that UTF-8 support breaks whenever
> LC_* vars are set but locales are not installed/configured, which
> would pretty much _always_ be the case when running a static-linked
> standalone binary on a non-musl-based system (where LC_* are probably
> set to something the main host libc recognizes).
> 
> One possibility if this behavior is problematic would be to only
> consider names without their own definitions as aliases for C.UTF-8
> when MUSL_LOCPATH is not set. However I think we'd need to see a
> strong motivation for doing that, since it seems like it would be
> worse behavior in some ways, especially when using LC_MESSAGES set to
> a language for which you don't have a locale installed.

I'm not an expert about locales to argue one way or the other.

Naively, I would think that this is somewhat problematic, because a best-behaving program (one that checks set locale's return code for errors) has no way to warn the user that he/she used an invalid locale.

Perhaps a work-around would be to handle it this way:
if an invalid (non-existing) locale is given in LC_* env vars, setlocale(LC_ALL,"") should return NULL (indicating an error), then all other invocations of setlocale(LC_*,NULL) would return the "C.UTF-8" indicator. This would allow detecting the error, but not affect further processing (if invalid locales are already an alias to C.UTF-8). This seems to match other OSes/libcs which return fixed "C" in such cases.

The reason for such check is that it is common user mistake to specify non-existing locales, then be confused by the seemingly incorrect results. Allowing a program to detect incorrect locales is a good mitigation.

I'll side-step the non-UTF-8 locales (which would be a problem in the current musl auto-aliasing to UTF-8), and show one possible case where silent aliasing leads to incorrect results.

consider the following UTF-8 string:
   M N Ñ O P Y Z Æ Ø Å
(which includes Spanish eñe and the last three letters in the Swedish alphabet).
When sorting with locale-aware programs, different locales should give different collation orders (e.g. es_ES.UTF-8 vs sv_FI.UTF-8).

To reproduce:
  A='\116\n\303\221\n\117\n\120\n\131\n\132\n\303\205\n\303\204\n\303\226\n'
  printf "$A" | LC_ALL=sv_FI.UTF-8 sort
  printf "$A" | LC_ALL=es_ES.UTF-8 sort

If a user has a typo in the locale name (e.g. sv_SV.UTF-8), there's no way for a program to detect it, and he will get unexpected ordered results.

GNU coreutils' 'sort' program added a --debug option to help user diagnose such issues.
On Linux with glibc, this will be the output:

  $ printf "$A" | LC_ALL=es_ES.UTF-8 sort --debug > /dev/null
  sort: using ‘es_ES.UTF-8’ sorting rules

  $ printf "$A" | LC_ALL=sv_FI.UTF-8 sort --debug > /dev/null                             
  sort: using ‘sv_FI.UTF-8’ sorting rules

  $ printf "$A" | LC_ALL=sv_SV.UTF-8 sort --debug > /dev/null                             
  sort: using simple byte comparison 

  $ printf "$A" | LC_ALL=foobar sort --debug > /dev/null                                   
  sort: using simple byte comparison

The last two messages ("simple byte") is the hint that the locale is invalid, and sort will does not use it.

On Alpine (linux + musl), there's no way to detect such case:

  $ printf "$A" | LC_ALL=sv_FI.UTF-8 gsort --debug > /dev/null
  gsort: using ‘sv_FI.UTF-8’ sorting rules

  $ printf "$A" | LC_ALL=sv_SV.UTF-8 gsort --debug > /dev/null
  gsort: using ‘sv_SV.UTF-8’ sorting rules

  $ printf "$A" | LC_ALL=foobar gsort --debug > /dev/null
  gsort: using ‘foobar’ sorting rules


regards,
 - assaf