Re: invalid characters and multi-byte [x-y] ranges

[Stephane Chazelas <stephane.chazelas@gmail.com>]

2015-09-03 10:00:37 +0100, Peter Stephenson:
> On Thu, 3 Sep 2015 00:07:11 +0100
> Stephane Chazelas <stephane.chazelas@gmail.com> wrote:
> > is this (in a UTF-8 locale):
> > 
> > $ zsh -c $'[[ \xcc = [\uaa-\udd] ]]' && echo yes
> > yes
> > 
> > expected or desirable?
> 
> This comes from the function charref() in pattern.c.  We discover the
> sequence is incomplete / invalid and don't know what to do with it, so we
> simply treat the single byte as a character:
> 
> 	return (wchar_t) STOUC(*x);
> 
> (the macro ensures we get an unsigned value to cast).  Typically this
> will do what you see (though wchar_t isn't guaranteed to have that
> property).
> 
> I'm not sure what else to do here.  The function is used all over the
> pattern code so anything other than tweak the code locally to return
> another character (what?) is horrific to get consistent.  We don't want
> [[ $'\xcc' = $'\xdd' ]] to succeed, but ideally we do want [[ $'\xcc' =
> $'\xcc' ]] to succeed comparing raw bytes (we're not morally forced to
> do that in a UTF-8 locale, I don't think, but it wouldn't be very
> helpful if it didn't work).
> 
> If wchar_t is 32 bits (the only place where it wasn't used to be Cygwin
> but I think that's changed) we could cheat by adding (wchar_t)0x7FFFFF00
> to it... that would fix your problem and (I hope) keep the two above
> working, and minimsie the likelihood of generating a valid character...
> that's about the least horrific I can come up with.
[...]

There was a related discussion not so long ago on the Austin
group ML (zsh was also mentioned there) on a related subject:

http://thread.gmane.org/gmane.comp.standards.posix.austin.general/11059/focus=11118
(the whole discussion started at
http://thread.gmane.org/gmane.comp.standards.posix.austin.general/11059/focus=11098 is interesting)

and see:
https://www.python.org/dev/peps/pep-0383/

A discussed approach there was to internally represent bytes not
forming part of a valid character as code points in the range
D800-DFFF (specifically DC80 DCFF for bytes 0x80 to 0xff) (those
code points are reserved in Unicode for UTF-16 surrogates and are *not*
characters, in particular the byte-sequence that
would be the UTF-8 encoding of a 0xD800 for example (ed a0 80)
would not form a valid character so be internally represented as 
DCED DCA0 DC80.

I beleive that's what python3 does.

-- 
Stephane