Re: [PATCH] zle_refresh multibyte fix

[Andrey Borzenkov <arvidjaar@newmail.ru>]

On Wednesday 23 February 2005 17:57, Peter Stephenson wrote:
>
> > Actually I find wc stuff very easy and suitable for using as internal
> > representation in zsh core. But this is separate topic.
>
> Apart from the inefficiency of extending every byte that comes into the
> shell into (typically) a four-byte integer, we can't rely on input and
> output bytes being a valid wide character in the current locale at all.
> I think the shell has to handle arbitrary strings of bytes without
> mutilating them.  Consider, for example:
>
>   # Pass secret byte to my utility
>   my_utility $'\xff'
>
> (or any other string you like, the only point being that it isn't a
> valid multibyte character string).  I don't see why we should
> arbitrarily decide that doesn't work because it doesn't convert to a
> wide character.  It will simply break far too many things.
>

I do not have easy answer, but what would be semantic of

- regexps ([[:print:]] et al.)?
- $foo[n,m] for scalar?
- Upper/Lower conversion?
- comparison (collating)?

apparently at least bash treats things byte-oriented:

bash-3.00$ echo к$(zsh -c "echo $'\xff\xff\xff'")  | xxd
0000000: d0ba ffff ff0a                           ......
bash-3.00$ echo к$(zsh -c "echo $'\xff\xff\xff'")  | { read foo; case "$foo" 
in ???? ) echo yes;; *   ) echo no; esac; }
no

but

bash-3.00$ zsh -c "echo $'\xff\xff\xff'"  | { read foo; case "$foo" in ??? ) 
echo yes;; *   ) echo no; esac; }
yes

and has definite problem editing non-ASCII input line (actually the same as 
zsh had - cursor position is wrong).

But it is hardly acceptable for zsh. It depends (completion code in the first 
place) too much on correct character handling.

OTOH does mb -> wc -> mb  preserves original byte sequence? It is true for 
UTF-8 (at least in current revision), but I am not sure that it is true for 
arbitrary encoding.

-andrey