Re: [ Guidance ] Potential New Routines; Requesting Help

[Rich Felker <dalias@libc.org>]

On Wed, Dec 25, 2019 at 09:07:05PM +0100, Florian Weimer wrote:
> * JeanHeyd Meneide:
> 
> >      I hope this e-mail finds you doing well this Holiday Season! I am
> > interested in developing a few fast routines for text encoding for
> > musl after the positive reception of a paper for the C Standard
> > related to fast conversion routines:
> >
> >      https://thephd.github.io/vendor/future_cxx/papers/source/C%20-%20Efficient%20Character%20Conversions.html
> 
> I'm somewhat concerned that the C multibyte functions are too broken
> to be useful.  There is a at least one widely implemented character
> set (Big5 as specified for HTML5) which does not fit the model implied
> by the standard.  Big5 does not have shift states, but a C
> implementation using UTF-32 for wchar_t has to pretend it has because
> correct conversion from Unicode to Big5 needs lookahead and cannot be
> performed one character at a time.

I don't think this can be modeled with shift states. C explicitly
forbids a stateful wchar_t encoding/multi-wchar_t-characters. Shift
states would be meaningful for the other direction.

In any case I don't think it really matters. There are no existing
implementations with this version of Big5 (with the offending HKSCS
characters included) as the locale charset, since it can't work, and
there really is no good reason to be adding *new* locale encodings.
The reason we (speaking of the larger community; musl doesn't) have
non-UTF-8 locales is legacy compatibility for users who need or insist
on keeping them.

If there really is an insistence on using this version of Big5, the
characters should be added to Unicode as <compat> characters so that
there's an unambiguous one-to-one correspondence, and the people who
care about it working should take responsibility for doing that.

> This would at least affect the proposed c8rtomb function.
> 
> I posted a brief review of the problematic charsets in glibc here:
> 
>   <https://sourceware.org/ml/libc-alpha/2019-05/msg00079.html>

I've read it but seemingly not in enough detail to gather what parts
are relevant to this conversation.

Rich