Possible bug with bindkey and Unicode characters in 5.2?

Possible bug with bindkey and Unicode characters in 5.2?

[Raymond W. Ko]

Hello, I am using Arch Linux, and zsh 5.2 recently landed in stable. I 
then discovered that some of my bindkeys with Unicode characters broke. 
I am basically using http://oremacs.com/2015/02/14/semi-xmodmap/ to 
generate unicode characters, but that should not be relevant.

Here is the relevant part of my .zshrc:

bindkey -s θ '"\C-b"'
bindkey -s ω "'\C-b'"
bindkey -s υ ">\C-b<"
bindkey -s σ "}\C-b{"
bindkey -s φ ")\C-b("
bindkey -s φ "asdf"
bindkey -s ρ "]\C-b["

On zsh 5.1 and 5.0.1, this works as expected. However, in 5.2 there 
seems to be a pause for half a second before a question mark appears. I 
have tested this on mintty, xterm and urxvt, and Linux and Cygwin, so I 
don't think this is terminal specific. Any ideas? Downgrading back to 
5.1 works for now.

Re: Possible bug with bindkey and Unicode characters in 5.2?

[Mikael Magnusson]

On Mon, Dec 7, 2015 at 9:47 PM, Raymond W. Ko <raymond.w.ko@gmail.com> wrote:
> Hello, I am using Arch Linux, and zsh 5.2 recently landed in stable. I then
> discovered that some of my bindkeys with Unicode characters broke. I am
> basically using http://oremacs.com/2015/02/14/semi-xmodmap/ to generate
> unicode characters, but that should not be relevant.
>
> Here is the relevant part of my .zshrc:
>
> bindkey -s θ '"\C-b"'
> bindkey -s ω "'\C-b'"
> bindkey -s υ ">\C-b<"
> bindkey -s σ "}\C-b{"
> bindkey -s φ ")\C-b("
> bindkey -s φ "asdf"
> bindkey -s ρ "]\C-b["
>
> On zsh 5.1 and 5.0.1, this works as expected. However, in 5.2 there seems to
> be a pause for half a second before a question mark appears. I have tested
> this on mintty, xterm and urxvt, and Linux and Cygwin, so I don't think this
> is terminal specific. Any ideas? Downgrading back to 5.1 works for now.

e1c0a947cc845c71dd844db44016d07922cfcefa is the first bad commit
commit e1c0a947cc845c71dd844db44016d07922cfcefa
Author: Peter Stephenson <p.w.stephenson@ntlworld.com>
Date:   Fri Sep 11 21:40:45 2015 +0100

    Read full multibyte string early for self-insert


You'd need to revert both
8a818381cfa843f6bdc25e3517500da31d63c282
e1c0a947cc845c71dd844db44016d07922cfcefa
and then it works fine again. I'll also note that pasting multibyte
text still works with my simple bracketed-paste-url-magic widget with
those commits reverted.

-- 
Mikael Magnusson

Re: Possible bug with bindkey and Unicode characters in 5.2?

[Peter Stephenson]

On Mon, 07 Dec 2015 23:29:41 +0100
Mikael Magnusson <mikachu@gmail.com> wrote:
> On Mon, Dec 7, 2015 at 9:47 PM, Raymond W. Ko <raymond.w.ko@gmail.com> wrote:
> > bindkey -s θ '"\C-b"'
> > bindkey -s ω "'\C-b'"
> > bindkey -s υ ">\C-b<"
> > bindkey -s σ "}\C-b{"
> > bindkey -s φ ")\C-b("
> > bindkey -s φ "asdf"
> > bindkey -s ρ "]\C-b["
> >
> > On zsh 5.1 and 5.0.1, this works as expected. However, in 5.2 there seems to
> > be a pause for half a second before a question mark appears.
>
> e1c0a947cc845c71dd844db44016d07922cfcefa is the first bad commit
> commit e1c0a947cc845c71dd844db44016d07922cfcefa
> Author: Peter Stephenson <p.w.stephenson@ntlworld.com>
> Date:   Fri Sep 11 21:40:45 2015 +0100
> 
>     Read full multibyte string early for self-insert

OK, I can reproduce this using the accented chracters.  AltGr-; e gives
me é and if I do

bindkey -s é 'foo'

then typing é has the effect reported.  I'll take a look.  It's not
specific to "-s".  I suspect it's getting the first octet and the second
is going somewhere else.  The change isn't quoted supposed to effect
non-self-insert bindings at all.

pws

Re: Possible bug with bindkey and Unicode characters in 5.2?

[Peter Stephenson]

On Mon, 07 Dec 2015 15:47:16 -0500
"Raymond W. Ko" <raymond.w.ko@gmail.com> wrote:
> bindkey -s θ '"\C-b"'
> bindkey -s ω "'\C-b'"
> bindkey -s υ ">\C-b<"
> bindkey -s σ "}\C-b{"
> bindkey -s φ ")\C-b("
> bindkey -s φ "asdf"
> bindkey -s ρ "]\C-b["
> 
> On zsh 5.1 and 5.0.1, this works as expected. However, in 5.2 there 
> seems to be a pause for half a second before a question mark appears.

Yes, this is a general problem.  This fix should be good enough.

I think there will be slight hiccups when there are a mixture of key
bindings for multibyte sequences and self-insert characters beginning
with the same bytes, but that's inevitable to some extent --- this should
certainly be no worse than before.  Bart ensured that even if we do
eventually need a self-insert it will work the way it always did, which
was fine in most cases, just a bit icky if you really needed the full
character to be inserted in good time.

(A volunteer to pick up ZLE testing would be useful.  Felix did quite a
lot of work on this some time ago which no one ever had the time to
merge.)

pws

diff --git a/Src/Zle/zle_keymap.c b/Src/Zle/zle_keymap.c
index 069580f..d6d116b 100644
--- a/Src/Zle/zle_keymap.c
+++ b/Src/Zle/zle_keymap.c
@@ -1503,7 +1503,7 @@ getkeymapcmd(Keymap km, Thingy *funcp, char **strp)
 		    f->widget->flags & ZLE_VIOPER);
 #ifdef MULTIBYTE_SUPPORT
 	    if ((f == Th(z_selfinsert) || f == Th(z_selfinsertunmeta)) &&
-		!lastchar_wide_valid) {
+		!lastchar_wide_valid && !ispfx) {
 		int len;
 		VARARR(char, mbc, MB_CUR_MAX);
 		ZLE_INT_T inchar = getrestchar(lastchar, mbc, &len);