zsh bug: isearch doesn't support unicode properly

zsh bug: isearch doesn't support unicode properly

[Yichao Yu]

Hi,

I've got a problem when typing unicode characters in the zsh's history search.

When trying to use unicode characters in isearch (the default binding
for ^S and ^R), some of the unicode characters are not recognized
correctly. For example, (with utf-8 encoding), when typing "重新安装"
(utf-8 string: \xe9\x87\x8d\xe6\x96\xb0\xe5\xae\x89\xe8\xa3\x85)
what's recognized by zsh is actually "駭涰宩裥" (utf-8 string:
\xe9\xa7\xad\xe6\xb6\xb0\xe5\xae\xa9\xe8\xa3\xa5e). It seems that the
fifth bit in one byte is flipped in come cases.

I'm not yet sure what's wrong with these characters but at least the
meta character processing looks suspicious.

Yichao Yu

Re: zsh bug: isearch doesn't support unicode properly

[Peter Stephenson]

On Mon, 28 Oct 2013 11:42:22 -0400
Yichao Yu <yyc1992@gmail.com> wrote:
> I've got a problem when typing unicode characters in the zsh's history search.
> 
> When trying to use unicode characters in isearch (the default binding
> for ^S and ^R), some of the unicode characters are not recognized
> correctly. For example, (with utf-8 encoding), when typing "重新安装"
> (utf-8 string: \xe9\x87\x8d\xe6\x96\xb0\xe5\xae\x89\xe8\xa3\x85)
> what's recognized by zsh is actually "駭涰宩裥" (utf-8 string:
> \xe9\xa7\xad\xe6\xb6\xb0\xe5\xae\xa9\xe8\xa3\xa5e). It seems that the
> fifth bit in one byte is flipped in come cases.
> 
> I'm not yet sure what's wrong with these characters but at least the
> meta character processing looks suspicious.

Yes --- how about this?

diff --git a/Src/Zle/zle_utils.c b/Src/Zle/zle_utils.c
index d0e7b55..b84d253 100644
--- a/Src/Zle/zle_utils.c
+++ b/Src/Zle/zle_utils.c
@@ -145,6 +145,7 @@ zlecharasstring(ZLE_CHAR_T inchar, char *buf)
 		ptr2--;
 	    }
 	    *ptr = Meta;
+	    ptr[1] ^= 32;
 	    ret++;
 	}
 
pws

Re: zsh bug: isearch doesn't support unicode properly

[Yichao Yu]

>> I'm not yet sure what's wrong with these characters but at least the
>> meta character processing looks suspicious.
>
> Yes --- how about this?
>
> diff --git a/Src/Zle/zle_utils.c b/Src/Zle/zle_utils.c
> index d0e7b55..b84d253 100644
> --- a/Src/Zle/zle_utils.c
> +++ b/Src/Zle/zle_utils.c
> @@ -145,6 +145,7 @@ zlecharasstring(ZLE_CHAR_T inchar, char *buf)
>                 ptr2--;
>             }
>             *ptr = Meta;
> +           ptr[1] ^= 32;
>             ret++;
>         }

Yep, patched it on the 5.0.2 release, tested with a few inputs that
used to have problems and it seems to fix the problem.

Thanks for the fast patch.

BTW, will this be included in the next release (5.0.3?). (Just want to
know if I still need to compile zsh myself on the next ArchLinux
update)

>
> pws

Re: zsh bug: isearch doesn't support unicode properly

[Peter Stephenson]

On Mon, 28 Oct 2013 13:53:41 -0400
Yichao Yu <yyc1992@gmail.com> wrote:
> BTW, will this be included in the next release (5.0.3?). (Just want to
> know if I still need to compile zsh myself on the next ArchLinux
> update)

Yes, I've commited it.

pws