From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/8583 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: First feedback on new C locale problems Date: Mon, 28 Sep 2015 15:34:00 -0400 Message-ID: <20150928193359.GE17773@brightrain.aerifal.cx> References: <20150901063235.GA18457@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: ger.gmane.org 1443468863 22864 80.91.229.3 (28 Sep 2015 19:34:23 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 28 Sep 2015 19:34:23 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-8595-gllmg-musl=m.gmane.org@lists.openwall.com Mon Sep 28 21:34:18 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1ZgeBj-0007c5-JS for gllmg-musl@m.gmane.org; Mon, 28 Sep 2015 21:34:15 +0200 Original-Received: (qmail 9888 invoked by uid 550); 28 Sep 2015 19:34:13 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 9852 invoked from network); 28 Sep 2015 19:34:12 -0000 Content-Disposition: inline In-Reply-To: <20150901063235.GA18457@brightrain.aerifal.cx> User-Agent: Mutt/1.5.21 (2010-09-15) Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:8583 Archived-At: I'm revisiting this thread because returning "UTF-8" for nl_langinfo(CODESET) in the C locale has seriously broken software using GNU regex (e.g. GNU sed) and seems unsafe in general. What's happening is that part of the code is using mbrtowc, and part is doing its own UTF-8 handling based on concluding from nl_langinfo(CODESET) that the locale is UTF-8 based. While there's no actual requirement in the standard for this to work, it seems reasonable for applications to expect that it works, and it presumably works on all existing implementations including all releases of musl (breakage is from commit 844212d94f582c4e3c5055e0a1524931e89ebe76, not yet in a release), and I'd really rather not make the real-world situation for handling UTF-8 _worse_ for applications. On Tue, Sep 01, 2015 at 02:32:35AM -0400, Rich Felker wrote: > So far I've gotten 2 reports of things breaking from the new C locale. > The first was in Alpine: > > gpg-connect-agent: conversion from 'utf-8' to 'UTF-8-CODE-UNITS' not available > > and turned out to be caused by --disable-nls omitting setlocale, > leading to nl_langinfo(CODESET) requesting the C locale's codeset > name. This could be fixed by making iconv support "UTF-8-CODE-UNITS" > and do something reasonable with it, but the second issue was uglier. Alpine now has a patch to call setlocale even when --disable-nls is used, and I think this is the right behavior. It's unreasonable to ever call nl_langinfo(CODESET) if you don't call setlocale or newlocale/uselocale. This patch should probably be upstreamed. Alternatively, they could remove the iconv code and pass strings through without any conversion when NLS is disabled, but that's probably a bad idea. > In Void Linux: > > help2man: > Unknown encoding 'UTF-8-CODE-UNITS' at /usr/bin/help2man line 56. > (https://github.com/voidlinux/void-packages/issues/2425) > > Oddly this one did not affect Alpine, for the same reason the first > one did: Alpine has gettext support turned off, and help2man omits the > offending code: > > http://anonscm.debian.org/cgit/users/bod/help2man.git/tree/help2man.PL?id=9ce0caa4cf164261ddde3fe987a260f5ba0dd558#n117 > > (which is overriding the system locale with "C" by default) when > gettext support is disabled. Being that this is Perl code and it's > passing the charset name to Perl's conversion functions, we can't just > work around this by adding a new charset alias to iconv. > > Note that fixing the broken programs to call setlocale properly and > honor the user's locale would make them work in the normal case, but > they would break again if the user explicitly invoked them with > LC_CTYPE=C. This remains the case. For all practical purposes, nl_langinfo(CODESET) must return a string which is a "well-known" character encoding name. This means there are exactly two choices: "UTF-8" or "ASCII". Neither is ideal. Returning "UTF-8" to the application misrepresents that multibyte character processing is active, and we've seen actual breakage (GNU regex) with this. GNU regex could be fixed by also checking MB_CUR_MAX here: http://git.savannah.gnu.org/cgit/gnulib.git/tree/lib/regcomp.c#n903 Actually they already do that in the (g)libc-internal code path, but not as part of gnulib. But in general, I'm worried that it doesn't make sense to demand that applications do this (and essentially create a situation where "UTF-8" can have two different meanings, i.e. "UTF-8 multibyte characters" and "nominal UTF-8 processed in units of bytes"). The other option, "ASCII", is also imperfect but perhaps better. The only place it's inconsistent is that iconv with "ASCII" as the in_charset would give EILSEQ for high bytes whereas mbrtowc would accept them and successfully round-trip them. But in conveying to applications the sense of "you're intentionally using a restricted character-set environment and ASCII is all that you can meaningfully use", it's accurate. It certainly doesn't permit any erroneous usage or misinterpretation of data. > Anyway, what I suspect is that we're going to find a fair number of > programs are calling nl_langinfo(CODESET) without actually having set > the locale properly. As long as they're not using multibyte functions > in libc to process text, failing to have called setlocale is not such > a bad thing; they can do character processing themselves if they know > the intended encoding, using iconv or native UTF-8 code or whatever. > And the situation we've got right now is that, despite best efforts > not to impact users who don't intentionally _try_ to get a byte-based > C locale, this functionality is causing actual regressions in musl's > promise of "always UTF-8". I suspect this problem will resurface, but at least we can find the affected applications and get them fixed. Does this sound reasonable? Rich