OK, i am busy on school these days. I read the mailing lists again, and i clean up. These are all remaining issues we need to solve since previous discussion:

1. about zero msgid1, i can prove that glibc will fallback to no translations. It's equal to printf(""), so this should be ok:

@@ -120,8 +122,9 @@

+ if (!msgid1) goto notrans;

> but it should be a separate patch since it's an independent

change.

(added in the head of dcngettext(), ill send a new standalone mail for this, but it's also included in this patch, be careful)

>But if the locale name is explicitly
non-UTF-8 like "zh_CN.GBK", we could opt to reject it without breaking
anything, and this may give users better feedback about what's going
wrong if they have such settings when ssh'ing into a musl-based
system.

About the .GBK(and any other non-UTF8 charsets), i ignore them by treating them as C.UTF-8, do we need to be more strict?

--- musl-1.1.16/src/locale/locale_map.c 2017-01-01 03:27:17.000000000 +0000

+++ musl-1.1.16/src/locale/locale_map.c 2017-01-01 03:27:17.000000000 +0000

@@ -46,7 +46,8 @@

if (val[0]=='.' || val[n]) val = "C.UTF-8";

int builtin = (val[0]=='C' && !val[1])

|| !strcmp(val, "C.UTF-8")

- || !strcmp(val, "POSIX");

+ || !strcmp(val, "POSIX")

+ || strcmp(__strchrnul(val, '.'), ".UTF-8");

if (builtin) {

if (cat == LC_CTYPE && val[1]=='.')

>The autoconf text for gettext is supposed to be getting fixed not to

do that anymore, but I'm not sure what the progress on upstreaming it
is.

It's just a workaround before they handle it, and i am not going to change anything in musl, just a description. I only patched myself.

> Support for non-UTF-8 .mo files won't be added.

> msgfmt just needs to be fixed not to produce

non-UTF-8 output.

I agreed with you, so then i hope '.UTF-8' could be kept. Rather than stripping it as i thought before, '.UTF-8' should be kept until the code went into dcngettext(). What if the .mo files are downloaded from www? Or what if it's pre-generated in the releases of programs? (i guess that's why vim gave me GBK set, it must be pre-generated)

And even with msgfmt generating UTF-8 outputs, what if programs still name the dir as zh_CN.UTF-8 instead of simply zh_CN? You can't say it's wrong, right? It's their preference how to name it.

It's necessary for those who have a full name like zh_CN.UTF-8 instead of zh_CN. This's what i am trying to express now.

2017-02-14 1:12 GMT+08:00 Rich Felker <dalias@libc.org>:

On Mon, Feb 13, 2017 at 10:06:49PM +0800, He X wrote:
> no, it's on musl, i just tested it with my patches, with vim, stripping
> will lead to unknown characters.

That's not a matter of the locale being non-UTF-8 (it's UTF-8) but of
the application doing something broken. The locale is UTF-8 because
nl_langinfo(CODESET) says it is and because mb/wc conversion functions
process UTF-8. That's what it means for the locale to be UTF-8.

> I mean, .mo files under zh_CN/ of vim is GBK set, while zh_CN/ of other
> apps is UTF-8 set, that meas there may be other apps like vim, we should be
> more cautious, add a check before map the .mo files, and fail non-UTF8 set
> in setlocale.

All musl locale files are required to be UTF-8. If an application has
translation files that are not UTF-8, they're not usable. This could
be fixed in the application or by using a fixed version of msgfmt that
converts to UTF-8 before producing the .mo file.

> Btw, _nl_msg_cat_cntr & _nl_domain_bindings will block apps compiling with
> the native intl of musl, and after i added a dump for these two symbols,

The autoconf text for gettext is supposed to be getting fixed not to
do that anymore, but I'm not sure what the progress on upstreaming it
is.

> gnu tar showed me segfaults, because he passed a zero msgid1 causing
> __mo_lookup segfault, we should add a check in dcngettext to avoid it(if
> (!msgid1) goto notrans;):
>
> #2 0x00007ffff7d82a6f in dcngettext (domainname=0x6737a0 "tar",
> msgid1=0x0, msgid2=0x0, n=1,
> category=5) at src/locale/dcngettext.c:211

Is it expecting gettext to return a null pointer in this case, or to
return something else (like the "header", i.e. the translation of "")?
I think it's acceptable to change this behavior as long as we do it
right, but it should be a separate patch since it's an independent
change.

Rich