Re: Re: a bug in bindtextdomain() and strip '.UTF-8'

[Rich Felker <dalias@libc.org>]

On Mon, Feb 13, 2017 at 10:06:49PM +0800, He X wrote:
> no, it's on musl, i just tested it with my patches, with vim, stripping
> will lead to unknown characters.

That's not a matter of the locale being non-UTF-8 (it's UTF-8) but of
the application doing something broken. The locale is UTF-8 because
nl_langinfo(CODESET) says it is and because mb/wc conversion functions
process UTF-8. That's what it means for the locale to be UTF-8.

> I mean, .mo files under zh_CN/ of vim is GBK set, while zh_CN/ of other
> apps is UTF-8 set, that meas there may be other apps like vim, we should be
> more cautious, add a check before map the .mo files, and fail non-UTF8 set
> in setlocale.

All musl locale files are required to be UTF-8. If an application has
translation files that are not UTF-8, they're not usable. This could
be fixed in the application or by using a fixed version of msgfmt that
converts to UTF-8 before producing the .mo file.

> Btw, _nl_msg_cat_cntr & _nl_domain_bindings will block apps compiling with
> the native intl of musl, and after i added a dump for these two symbols,

The autoconf text for gettext is supposed to be getting fixed not to
do that anymore, but I'm not sure what the progress on upstreaming it
is.

> gnu tar showed me segfaults, because he passed a zero msgid1 causing
> __mo_lookup segfault, we should add a check in dcngettext to avoid it(if
> (!msgid1) goto notrans;):
> 
>  #2  0x00007ffff7d82a6f in dcngettext (domainname=0x6737a0 "tar",
> msgid1=0x0, msgid2=0x0, n=1,
>     category=5) at src/locale/dcngettext.c:211

Is it expecting gettext to return a null pointer in this case, or to
return something else (like the "header", i.e. the translation of "")?
I think it's acceptable to change this behavior as long as we do it
right, but it should be a separate patch since it's an independent
change.

Rich