From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/15069 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Florian Weimer Newsgroups: gmane.linux.lib.musl.general Subject: Re: [ Guidance ] Potential New Routines; Requesting Help Date: Thu, 26 Dec 2019 10:43:06 +0100 Message-ID: <87sgl7zal1.fsf@mid.deneb.enyo.de> References: <87zhfg185y.fsf@mid.deneb.enyo.de> <20191226021354.GE30412@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com Mime-Version: 1.0 Content-Type: text/plain Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="167406"; mail-complaints-to="usenet@blaine.gmane.org" Cc: JeanHeyd Meneide , musl@lists.openwall.com To: Rich Felker Original-X-From: musl-return-15084-gllmg-musl=m.gmane.org@lists.openwall.com Thu Dec 26 10:44:10 2019 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.89) (envelope-from ) id 1ikPgg-000hR8-8L for gllmg-musl@m.gmane.org; Thu, 26 Dec 2019 10:44:10 +0100 Original-Received: (qmail 22325 invoked by uid 550); 26 Dec 2019 09:44:07 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 22307 invoked from network); 26 Dec 2019 09:44:06 -0000 In-Reply-To: <20191226021354.GE30412@brightrain.aerifal.cx> (Rich Felker's message of "Wed, 25 Dec 2019 21:13:54 -0500") Xref: news.gmane.org gmane.linux.lib.musl.general:15069 Archived-At: * Rich Felker: > On Wed, Dec 25, 2019 at 09:07:05PM +0100, Florian Weimer wrote: >> * JeanHeyd Meneide: >> >> > I hope this e-mail finds you doing well this Holiday Season! I am >> > interested in developing a few fast routines for text encoding for >> > musl after the positive reception of a paper for the C Standard >> > related to fast conversion routines: >> > >> > https://thephd.github.io/vendor/future_cxx/papers/source/C%20-%20Efficient%20Character%20Conversions.html >> >> I'm somewhat concerned that the C multibyte functions are too broken >> to be useful. There is a at least one widely implemented character >> set (Big5 as specified for HTML5) which does not fit the model implied >> by the standard. Big5 does not have shift states, but a C >> implementation using UTF-32 for wchar_t has to pretend it has because >> correct conversion from Unicode to Big5 needs lookahead and cannot be >> performed one character at a time. > > I don't think this can be modeled with shift states. C explicitly > forbids a stateful wchar_t encoding/multi-wchar_t-characters. Shift > states would be meaningful for the other direction. The intent of the standard appears to be to support this as an extension. It's hard to tell because actual users of the interfaces with legacy charsets do not seem to be represented on the standards committee anymore (see the mblen behavioral change in C11 as evidence supporting this theory). > In any case I don't think it really matters. There are no existing > implementations with this version of Big5 (with the offending HKSCS > characters included) as the locale charset, since it can't work, and > there really is no good reason to be adding *new* locale encodings. Do you mean in musl? > The reason we (speaking of the larger community; musl doesn't) have > non-UTF-8 locales is legacy compatibility for users who need or insist > on keeping them. That is true. > If there really is an insistence on using this version of Big5, the > characters should be added to Unicode as characters so that > there's an unambiguous one-to-one correspondence, and the people who > care about it working should take responsibility for doing that. Yes, I was very surprised this wasn't done for TSCII and HKSCS/Big5. I think even for Big5, it would solve the issue because the decoding process only needs to look at a single multibyte character at the time (I may have suggested otherwise in the past). A succint description of what is going on for Big 5 is here: , under step 3.3. The conversion is actually fairly simple, it's just hard to fit it into the C interfaces. >> This would at least affect the proposed c8rtomb function. >> >> I posted a brief review of the problematic charsets in glibc here: >> >> > > I've read it but seemingly not in enough detail to gather what parts > are relevant to this conversation. It names a few character sets that have fake shift states because the C interfaces cannot otherwise be used with them. Some of the new interfaces are problematic in this context (whether or not UTF-32 is used for wchar_t). I think new interfaces should be compatible with existing implementation practice. The other thing I found surprising is that there are no ASCII-transparent charsets with traditional shift states in glibc. The ASCII-transparent charsets with shift states have these fake shift states.