clarification on (#U) in pattern matching.

clarification on (#U) in pattern matching.

[Stephane Chazelas]

$ set -o extendedglob
$ a='Stéphane€'
$ print -rn -- ${a//(#U)?} | hd
00000000  a9 82 ac                                          |...|
00000003

It seems that with (#U) (and here in a locale using UTF-8 as
charmap), ? with (#U) matches only on the first byte of
multibyte characters. Is that how it's meant to be?

$ print -r -- ${a//(#m)?/[$MATCH]}
[S][t][é][p][h][a][n][e][€]
$ print -r -- ${a//(#Um)?/[$MATCH]}
[S][t][�]�[p][h][a][n][e][�]��

Also

[[ $'\ue9' = (#U)*$'\xa9'* ]] returns true (and doesn't without
(#U), but:

print -r -- ${a//(#U)$'\xa9'}

fails to remove it. But:

$ echo ${a//(#U)?$'\xa9'}
Stphane€

With set +o multibyte:

$ set +o multibyte
$ print -r -- ${a//(#m)?/[$MATCH]}
[S][t][�][�][p][h][a][n][e][�][�][�]

Where ? matches on each byte of those multi-byte characters.

The doc has:

> U
>     All characters are considered to be a single byte long.  The
>     opposite of u.  This overrides the MULTIBYTE option.

which is a bit ambiguous and may be interpreted as justifying
the current behaviour.

But I suspect that's because when ${var//pattern/replace}
resumes searching for the next pattern after the first one, it
starts at the next character instead of next byte, and the (#U)
applies to pattern matching but not to the
${var//pattern/replace} operator itself.

Thanks
Stephane

Re: clarification on (#U) in pattern matching.

[Peter Stephenson]

Sorry, this just went to Stephane.

pws

> On 07 February 2022 at 11:30 Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> 
> 
> > On 06 February 2022 at 08:42 Stephane Chazelas <stephane@chazelas.org> wrote:
> > $ set -o extendedglob
> > $ a='Stéphane€'
> > $ print -rn -- ${a//(#U)?} | hd
> > 00000000  a9 82 ac                                          |...|
> > 00000003
> > 
> > It seems that with (#U) (and here in a locale using UTF-8 as
> > charmap), ? with (#U) matches only on the first byte of
> > multibyte characters. Is that how it's meant to be?
> 
> I think what you're hitting is probably, as you suspected, a
> difference between the pattern matching code and the substitution
> code.  The underlying pattern matching really is byte by byte,
> but this doesn't force any substitution such as // to behave
> in the same way.  As far as I know, the MULTIBYTE option is
> the only higher level consistency measure we have.
> 
> I think there might be a parameter matching flag that you can
> also set that would help.  I'd have to look in more detail.
> 
> pws

Re: clarification on (#U) in pattern matching.

[Stephane Chazelas]

See also:

$ a=été
$ echo ${a%(#U)?}
été
$ echo ${a%%(#U)?}
été
$ echo ${a%(#U)$'\xa9'}
été
$ echo ${a%(#U)?*}
ét

Consistent with the other behaviours but maybe even more confusing.

-- 
Stephane

Re: clarification on (#U) in pattern matching.

[Peter Stephenson]

On 07 February 2022 at 12:15 Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> I think there might be a parameter matching flag that you can
> also set that would help.  I'd have to look in more detail.

On examination, no, I don't think there is an option at the parameter level,
as opposed to the pattern match level --- and I appreciate that this
distinction, while clear in the code, isn't likely to be so obvious to a
user staring at shell code, even if parameter substitution and pattern
matching are documented separately. The doc for parameters simplify refers
to the MULTIBYTE option.

There's a limited feature to turn off multibyte counting when calculating
widths for use in padding etc. I suppose in principle that could be extended
for pattern matching, but in practice (at least without a major rewrite) all
it would do is turn off the multibyte option locally --- probably carrying over
into pattern matching, so you'd at least get consistency between the two that
way round.

There isn't a sane way to propagate the (U) flag out of the pattern code
back up into the parameter level.

pws