From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/15064 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: Re: [ Guidance ] Potential New Routines; Requesting Help Date: Wed, 25 Dec 2019 21:13:54 -0500 Message-ID: <20191226021354.GE30412@brightrain.aerifal.cx> References: <87zhfg185y.fsf@mid.deneb.enyo.de> Reply-To: musl@lists.openwall.com Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="70195"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Mutt/1.5.21 (2010-09-15) Cc: JeanHeyd Meneide , musl@lists.openwall.com To: Florian Weimer Original-X-From: musl-return-15080-gllmg-musl=m.gmane.org@lists.openwall.com Thu Dec 26 03:14:16 2019 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.89) (envelope-from ) id 1ikIfI-000I9I-00 for gllmg-musl@m.gmane.org; Thu, 26 Dec 2019 03:14:16 +0100 Original-Received: (qmail 27656 invoked by uid 550); 26 Dec 2019 02:14:13 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 26614 invoked from network); 26 Dec 2019 02:14:12 -0000 Content-Disposition: inline In-Reply-To: <87zhfg185y.fsf@mid.deneb.enyo.de> Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:15064 Archived-At: On Wed, Dec 25, 2019 at 09:07:05PM +0100, Florian Weimer wrote: > * JeanHeyd Meneide: > > > I hope this e-mail finds you doing well this Holiday Season! I am > > interested in developing a few fast routines for text encoding for > > musl after the positive reception of a paper for the C Standard > > related to fast conversion routines: > > > > https://thephd.github.io/vendor/future_cxx/papers/source/C%20-%20Efficient%20Character%20Conversions.html > > I'm somewhat concerned that the C multibyte functions are too broken > to be useful. There is a at least one widely implemented character > set (Big5 as specified for HTML5) which does not fit the model implied > by the standard. Big5 does not have shift states, but a C > implementation using UTF-32 for wchar_t has to pretend it has because > correct conversion from Unicode to Big5 needs lookahead and cannot be > performed one character at a time. I don't think this can be modeled with shift states. C explicitly forbids a stateful wchar_t encoding/multi-wchar_t-characters. Shift states would be meaningful for the other direction. In any case I don't think it really matters. There are no existing implementations with this version of Big5 (with the offending HKSCS characters included) as the locale charset, since it can't work, and there really is no good reason to be adding *new* locale encodings. The reason we (speaking of the larger community; musl doesn't) have non-UTF-8 locales is legacy compatibility for users who need or insist on keeping them. If there really is an insistence on using this version of Big5, the characters should be added to Unicode as characters so that there's an unambiguous one-to-one correspondence, and the people who care about it working should take responsibility for doing that. > This would at least affect the proposed c8rtomb function. > > I posted a brief review of the problematic charsets in glibc here: > > I've read it but seemingly not in enough detail to gather what parts are relevant to this conversation. Rich