From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/11153 Path: news.gmane.org!.POSTED!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: Re: a bug in bindtextdomain() and strip '.UTF-8' Date: Sat, 18 Mar 2017 08:28:33 -0400 Message-ID: <20170318122833.GN1693@brightrain.aerifal.cx> References: <20170212023422.GE1520@brightrain.aerifal.cx> <20170213132816.GG1520@brightrain.aerifal.cx> <20170213171236.GI1520@brightrain.aerifal.cx> <20170317192749.GL1693@brightrain.aerifal.cx> <20170317193740.GM1693@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1489840138 2431 195.159.176.226 (18 Mar 2017 12:28:58 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sat, 18 Mar 2017 12:28:58 +0000 (UTC) User-Agent: Mutt/1.5.21 (2010-09-15) To: musl@lists.openwall.com Original-X-From: musl-return-11168-gllmg-musl=m.gmane.org@lists.openwall.com Sat Mar 18 13:28:53 2017 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1cpDTP-0007l7-CD for gllmg-musl@m.gmane.org; Sat, 18 Mar 2017 13:28:43 +0100 Original-Received: (qmail 24138 invoked by uid 550); 18 Mar 2017 12:28:46 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 24118 invoked from network); 18 Mar 2017 12:28:45 -0000 Content-Disposition: inline In-Reply-To: Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:11153 Archived-At: On Sat, Mar 18, 2017 at 07:34:58AM +0000, He X wrote: > > As discussed on irc, .charset suffixes should be dropped before the > loop even begins (never used in pathnames), and they occur before the > @mod, not after it, so the logic for dropping them is different. > > 1. drop .charset: Sorry for proposing it again, i forget this case after > around three weeks, as i said before, vim will generate three different .mo > files with different charset -> zh_CN.UTF-8.po, zh_CN.cp936.po, zh_CN.po. > In that case, dropping is to generate a lots of junk. > > I now found it's not a bug of msgfmt. That is charset is converted by: > iconv -f UTF-8 -t cp936 zh_CN.UTF-8.po | sed -e > 's/charset=utf-8/charset=gbk/ > ... So that means, charset and pathname is > decided by softwares, msgfmt does not do charset converting at all, just a > format-translator. (btw, iconv.c is from alpine) There are two things you seem to be missing: 1. musl does not, and won't, support non-UTF-8 locales, so there is no point in trying to load translations for them. Moreover, with the proposed changes to setlocale/locale_map.c, it will never be possible for the locale name to contain a . with anything other than UTF-8 (or, for compatibility, some variant like utf8) after it. So I don't see how there's any point in iterating and trying with/without .charset when the only possibilities are that .charset is blank, .UTF-8, or some misspelling of .UTF-8. In the latter case, we'd even have to do remapping of the misspellings to avoid having to have multiple dirs/symlinks. 2. From my perspective, msgfmt's production of non-UTF-8 .mo files is a bug. Yes the .po file can be something else, but msgfmt should be transcoding it at 'compile' time. There's at least one other change msgfmt needs for all features to work with musl's gettext -- expansion of SYSDEP strings to all their possible format patterns -- so I don't think it's a significant additional burden to ensure that the msgfmt used on musl-based systems outputs UTF-8. Of course software trying to do multiple encodings like you described will still install duplicate files unless patched, but any of them should work as long as msgfmt recoded them. In the mean time, distros can just patch the build process for software that's still installing non-UTF-8 locale files. AFAIK doing that is not a recommended practice even by the GNU gettext project, so the patches might even make it upstream. One thing we could do for robustness is check the .mo header at load time and, if it has a charset= specification with something other than UTF-8, reject it. I mainly suggest this in case the program is running on a non-musl system where a glibc-built version of the same program (e.g. vi) with non-UTF-8 .mo files is present and they're using the same textdomain dir (actually unlikely since prefix should be different). But if we do this it should be a separate patch because it's a separate functional change. Rich