Re: invalid characters and multi-byte [x-y] ranges

[Peter Stephenson <p.stephenson@samsung.com>]

On Thu, 3 Sep 2015 00:07:11 +0100
Stephane Chazelas <stephane.chazelas@gmail.com> wrote:
> is this (in a UTF-8 locale):
> 
> $ zsh -c $'[[ \xcc = [\uaa-\udd] ]]' && echo yes
> yes
> 
> expected or desirable?

This comes from the function charref() in pattern.c.  We discover the
sequence is incomplete / invalid and don't know what to do with it, so we
simply treat the single byte as a character:

	return (wchar_t) STOUC(*x);

(the macro ensures we get an unsigned value to cast).  Typically this
will do what you see (though wchar_t isn't guaranteed to have that
property).

I'm not sure what else to do here.  The function is used all over the
pattern code so anything other than tweak the code locally to return
another character (what?) is horrific to get consistent.  We don't want
[[ $'\xcc' = $'\xdd' ]] to succeed, but ideally we do want [[ $'\xcc' =
$'\xcc' ]] to succeed comparing raw bytes (we're not morally forced to
do that in a UTF-8 locale, I don't think, but it wouldn't be very
helpful if it didn't work).

If wchar_t is 32 bits (the only place where it wasn't used to be Cygwin
but I think that's changed) we could cheat by adding (wchar_t)0x7FFFFF00
to it... that would fix your problem and (I hope) keep the two above
working, and minimsie the likelihood of generating a valid character...
that's about the least horrific I can come up with.

pws