Zsh changes on Ubuntu

Zsh changes on Ubuntu

[Thorsten Kampe]

Hi,

I'm using Zsh in connection with Fizsh bindings on Ubuntu 12.04. 
Fizsh has a very useful feature (similar to up-line-or-beginning-
search) to the effect that the cursor keys would do a "sub-string-
search". So if you typed ssh and used the cursor keys, it would 
search in the history for the next entry which contained ssh.

This feature doesn't work anymore in Ubuntu 13.10 and 14.04 (not sure 
about 12.10 and 13.04)

Zsh is 4.3.17 on Ubuntu 12.04. On Ubuntu 13.10 it is 5.0.2. 
Additionally there is now a zsh-common package. Fizsh is still the 
same version as on Ubuntu 12.04. Fizsh works fine on Cygwin (which 
also uses 5.0.2.

How can I troubleshoot this issue?

Thorsten

Re: Zsh changes on Ubuntu

[Frank Terbeck]

Hello,

let me put on my debian pkg-zsh team hat on for this one...

Thorsten Kampe wrote:
> I'm using Zsh in connection with Fizsh bindings on Ubuntu 12.04. 
> Fizsh has a very useful feature (similar to up-line-or-beginning-
> search) to the effect that the cursor keys would do a "sub-string-
> search". So if you typed ssh and used the cursor keys, it would 
> search in the history for the next entry which contained ssh.
>
> This feature doesn't work anymore in Ubuntu 13.10 and 14.04 (not sure 
> about 12.10 and 13.04)
>
> Zsh is 4.3.17 on Ubuntu 12.04. On Ubuntu 13.10 it is 5.0.2. 
> Additionally there is now a zsh-common package. Fizsh is still the 
> same version as on Ubuntu 12.04. Fizsh works fine on Cygwin (which 
> also uses 5.0.2.
>
> How can I troubleshoot this issue?

Since ubuntu is debian's downstream, I'm pretty sure this may be tied to
a change in debian's global zshrc: In pre-5.x packages, debian used a
rather broken approach trying to read escape sequences for special keys
from the terminfo database.

Starting with the package for 5.0.0 and onward¹, that was fixed by
setting the terminal zsh is running in into the correct mode for the
values from terminfo to be applicable while zsh's line editor (ZLE) is
running. This was done to make the setup work across as many different
terminals as possible.

If you had bound keys to hard-coded escape sequences that worked outside
of that mode, those may be invalid with that mode active. To bind
something to - say - the Up-Cursor key, you'd do this:

  bindkey "${key[Up]}" some-widgets-name

To get a list of valid keys for the ‘$key[]’ map:

  print ${(k)key}

We also added an entry to the package's NEWS file about this change².
The file also mentions a "Get out of my way, I know what I'm doing!"
variable, that serves as a way to disable the package's keyboard setup,
should you want to go a different route.


Regards, Frank

¹ http://anonscm.debian.org/gitweb/?p=collab-maint/zsh.git;a=blob;f=debian/zshrc;hb=HEAD
² http://anonscm.debian.org/gitweb/?p=collab-maint/zsh.git;a=blob;f=debian/NEWS;hb=HEAD

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Zsh changes on Ubuntu

[Thorsten Kampe]

* Frank Terbeck (Sat, 17 May 2014 12:29:58 +0200)
> 
> Since ubuntu is debian's downstream, I'm pretty sure this may be tied to
> a change in debian's global zshrc: In pre-5.x packages, debian used a
> rather broken approach trying to read escape sequences for special keys
> from the terminfo database.
> 
> Starting with the package for 5.0.0 and onward¹, that was fixed by
> setting the terminal zsh is running in into the correct mode for the
> values from terminfo to be applicable while zsh's line editor (ZLE) is
> running. This was done to make the setup work across as many different
> terminals as possible.
> 
> If you had bound keys to hard-coded escape sequences that worked outside
> of that mode, those may be invalid with that mode active. To bind
> something to - say - the Up-Cursor key, you'd do this:
> 
>   bindkey "${key[Up]}" some-widgets-name
> 
> To get a list of valid keys for the ‘$key[]’ map:
> 
>   print ${(k)key}
> 
> We also added an entry to the package's NEWS file about this change².
> The file also mentions a "Get out of my way, I know what I'm doing!"
> variable, that serves as a way to disable the package's keyboard setup,
> should you want to go a different route.

"export DEBIAN_PREVENT_KEYBOARD_CHANGES=yes" in .zshenv did the 
trick.

Thanks, Thorsten

Re: Zsh changes on Ubuntu

[Frank Terbeck]

Thorsten Kampe wrote:
> * Frank Terbeck (Sat, 17 May 2014 12:29:58 +0200)
>> Since ubuntu is debian's downstream, I'm pretty sure this may be tied to
>> a change in debian's global zshrc: In pre-5.x packages, debian used a
>> rather broken approach trying to read escape sequences for special keys
>> from the terminfo database.
>> 
>> Starting with the package for 5.0.0 and onward¹, that was fixed by
>> setting the terminal zsh is running in into the correct mode for the
>> values from terminfo to be applicable while zsh's line editor (ZLE) is
>> running. This was done to make the setup work across as many different
>> terminals as possible.
[...]
> "export DEBIAN_PREVENT_KEYBOARD_CHANGES=yes" in .zshenv did the 
> trick.

No need to export the parameter. No external process will need it, and
zshenv is read for every zsh-instance, that also reads zshrc.

Generally, I wouldn't advice people to set the parameter. Setting the
terminal into "keyboard-transmit-mode" (which is what the smkx terminfo
entry does) while the line editor is active is the right thing to do,
IMNSHO. Otherwise, you can't rely on values from $terminfo to be
correct. The parameter is really meant for people who know what they are
doing.

I'm just mentioning this for the innocent lurker, who might be wondering
what we're on about. I don't know of your background and you might
indeed know what you're doing. This is not meant to be offensive.

Regards, Frank

Re: Zsh changes on Ubuntu

[Thorsten Kampe]

* Frank Terbeck (Mon, 19 May 2014 14:03:30 +0200)
> 
> Thorsten Kampe wrote:
> > * Frank Terbeck (Sat, 17 May 2014 12:29:58 +0200)
> >> Since ubuntu is debian's downstream, I'm pretty sure this may be tied to
> >> a change in debian's global zshrc: In pre-5.x packages, debian used a
> >> rather broken approach trying to read escape sequences for special keys
> >> from the terminfo database.
> >> 
> >> Starting with the package for 5.0.0 and onward¹, that was fixed by
> >> setting the terminal zsh is running in into the correct mode for the
> >> values from terminfo to be applicable while zsh's line editor (ZLE) is
> >> running. This was done to make the setup work across as many different
> >> terminals as possible.
> [...]
> > "export DEBIAN_PREVENT_KEYBOARD_CHANGES=yes" in .zshenv did the 
> > trick.
> 
> No need to export the parameter. No external process will need it, and
> zshenv is read for every zsh-instance, that also reads zshrc.
> 
> Generally, I wouldn't advice people to set the parameter. Setting the
> terminal into "keyboard-transmit-mode" (which is what the smkx terminfo
> entry does) while the line editor is active is the right thing to do,
> IMNSHO. Otherwise, you can't rely on values from $terminfo to be
> correct. The parameter is really meant for people who know what they are
> doing.
> 
> I'm just mentioning this for the innocent lurker, who might be wondering
> what we're on about. I don't know of your background and you might
> indeed know what you're doing. This is not meant to be offensive.

No, I don"t know what I'm doing with that parameter. As far as I can 
see it reverts the keyboard handling back to Ubuntu 12.04 so it can't 
be that dramatic. And not exporting the parameter doesn't work with 
Fizsh.

Thorsten

Re: Zsh changes on Ubuntu

[Frank Terbeck]

Thorsten Kampe wrote:
[...]
> No, I don"t know what I'm doing with that parameter. As far as I can 
> see it reverts the keyboard handling back to Ubuntu 12.04 so it can't 
> be that dramatic.

No. Setting the parameter disables all additional keyboard setup in the
global zshrc. Hence the name. With it set, you get the default zsh
behaviour. It doesn't make any effort to support any special keys (like
PgUp, etc) whatsoever.

And that doesn't set smkx mode. IIRC, it was decided not to set smkx
mode by default, to avoid breaking existing configs, not because it
would be the wrong thing to do.

> And not exporting the parameter doesn't work with Fizsh.

Oh-kay. That makes me pretty happy, I don't have to look that. :-)

If it's just zsh, and zshenv sets a variable, that'll be set in zshrc
(which is read after zshenv). So the variable can just be accessed in
zshrc without any more effort. I have no idea what that package does,
that would break that.


Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Zsh changes on Ubuntu

[Axel Beckert]

Hi,

On Mon, May 19, 2014 at 10:32:49PM +0200, Thorsten Kampe wrote:
> > I'm just mentioning this for the innocent lurker, who might be wondering
> > what we're on about. I don't know of your background and you might
> > indeed know what you're doing. This is not meant to be offensive.
> 
> No, I don"t know what I'm doing with that parameter. As far as I can 
> see it reverts the keyboard handling back to Ubuntu 12.04 so it can't 
> be that dramatic. And not exporting the parameter doesn't work with 
> Fizsh.

In that case, it sounds like a bug in fizsh or at least the fizsh
package in Ubuntu for me.

In general I'd report that against fizsh in Ubuntu then. Seems not yet
reported: https://bugs.launchpad.net/ubuntu/+source/fizsh

Then again, the fiszh developer -- who also maintains the Debian
packages of fiszh -- sadly doesn't seem to have much interest in the
Debian/Ubuntu packages anymore as he released 1.0.5 in 2011 but never
uploaded packages for the past three releases to Debian or Ubuntu --
he only provides them on Sourceforge:

http://sourceforge.net/projects/fizsh/files/
http://packages.qa.debian.org/f/fizsh.html

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5

Re: Zsh changes on Ubuntu

[Guido van Steen]

[-- Attachment #1: Type: text/plain, Size: 3464 bytes --]

Hi Axel

It makes me happy to see your and Torsten's interest in my Debian package. I am the developer of Fizsh. I am also the Debian Maintainer of the Debian package, but since I am not a Debian Developer I do not have upload rights to Debian. 

> Then again, the fiszh developer -- who also maintains the Debian
> packages of fiszh -- sadly doesn't seem to have much interest in the
> Debian/Ubuntu packages anymore as he released 1.0.5 in 2011 but never
> uploaded packages for the past three releases to Debian or Ubuntu --

The above is not exactly true. I did upload the most recent version of Fizsh to Debian Mentors (see: https://lists.debian.org/debian-mentors/2011/10/msg00404.html). However, no Debian Developer showed any interest in this sponsorship request. 

Yet, since you are a Debian Developer, you may still sponsor the latest version of Fizsh (1.0.5) into Debian. 

After I finished version 1.0.5 almost three years ago, the package was 100% Lintian clean. Meanwhile Lintian might complain about some violations of modifications of the Debian Packaging Policy. I am willing to address such issues, on the condition that you will eventually sponsor the improved package. In that case I will also try to correct the issue reported by Torsten. I will further include the latest versions of "zsh-history-substring-search" and "zsh-syntax-highlighting". 

So, if you agree, let's do some work together, and have a new version of Fizsh uploaded to Debian 8/Jessie. 


Best wishes, 

Guido van Steen 

PS. I would not file a bug against the current Ubuntu package. This package is an old version, and it is an exact copy of the Debian package. If bugs are corrected, let's correct them in the most recent version, which is indeed available from Sourceforge (http://sourceforge.net/projects/fizsh/files/fizsh_1.0.5-1_all.deb/download). 

On Thursday, 22 May 2014, 19:10, Axel Beckert <abe@debian.org> wrote:
 

>
>
>Hi,
>
>
>On Mon, May 19, 2014 at 10:32:49PM +0200, Thorsten Kampe wrote:
>> > I'm just mentioning this for the innocent lurker, who might be wondering
>> > what we're on about. I don't know of your background and you might
>> > indeed know what you're doing. This is not meant to be offensive.
>> 
>> No, I don"t know what I'm doing with that parameter. As far as I can 
>> see it reverts the keyboard handling back to Ubuntu 12.04 so it can't 
>> be that dramatic. And not exporting the parameter doesn't work with 
>> Fizsh.
>
>In that case, it sounds like a bug in fizsh or at least the fizsh
>package in Ubuntu for me.
>
>In general I'd report that against fizsh in Ubuntu then. Seems not yet
>reported: https://bugs.launchpad.net/ubuntu/+source/fizsh
>
>Then again, the fiszh developer -- who also maintains the Debian
>packages of fiszh -- sadly doesn't seem to have much interest in the
>Debian/Ubuntu packages anymore as he released 1.0.5 in 2011 but never
>uploaded packages for the past three releases to Debian or Ubuntu --
>he only provides them on Sourceforge:
>
>http://sourceforge.net/projects/fizsh/files/
>http://packages.qa.debian.org/f/fizsh.html
>
>        Regards, Axel
>-- 
>,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
>: :' :  |  Debian Developer, ftp.ch.debian.org Admin
>`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
>  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5
>
>
>
>

Re: Zsh changes on Ubuntu

[Axel Beckert]

Hi Guido!

I'm Cc'ing zsh-users once more for explanations which may be
interesting for others, too. But I think that we should take the
remainder of the package upload sponsoring discussion off-list.

On Fri, May 23, 2014 at 12:59:22AM -0700, Guido van Steen wrote:
> It makes me happy to see your and Torsten's interest in my Debian
> package. I am the developer of Fizsh.

And you are reading this list, good! :-)

I planned to reproduce Torsten's issue on Debian and would probably
have reported it against the Debian package.

> > Then again, the fiszh developer -- who also maintains the Debian
> > packages of fiszh -- sadly doesn't seem to have much interest in the
> > Debian/Ubuntu packages anymore as he released 1.0.5 in 2011 but never
> > uploaded packages for the past three releases to Debian or Ubuntu --
> 
> The above is not exactly true. I did upload the most recent version
> of Fizsh to Debian Mentors (see:
> https://lists.debian.org/debian-mentors/2011/10/msg00404.html).
> However, no Debian Developer showed any interest in this sponsorship
> request. 

That's a pity. I must admit, I haven't checked the mailing-list
archives. As I see now I should have...

And since mentors.debian.net has been rewritten since then, old
uploads are no more shown on
http://packages.qa.debian.org/f/fizsh.html -- I would have noticed
that otherwise since more recent uploads do.

I would also have noticed if you would have commented on
https://bugs.debian.org/704863 -- posting such a comment into the BTS
causes probably more long-time awareness for sponsoring needs than a
single post on the mailing list. In the meanwhile there's also a
pseudo package called sponsorship-requests[1] and bug reports against
that package are shown by the recent how-can-i-help tool if the
according package. (Yes, that's all improvements that weren't there
back then.)

[1] https://bugs.debian.org/sponsorship-requests

> Yet, since you are a Debian Developer, you may still sponsor the
> latest version of Fizsh (1.0.5) into Debian. 

Please reupload to mentors (or any other place where I can download
it) and I'll review and likely sponsor the package.

> After I finished version 1.0.5 almost three years ago, the package
> was 100% Lintian clean. Meanwhile Lintian might complain about some
> violations of modifications of the Debian Packaging Policy. I am
> willing to address such issues, on the condition that you will
> eventually sponsor the improved package.

Thanks, appreciated!

> In that case I will also try to correct the issue reported by
> Torsten.

Much appreciated! Frank knows details as you may have noticed, so for
that you better ask him (or this list).

> I will further include the latest versions of
> "zsh-history-substring-search" and "zsh-syntax-highlighting". 

... which means it will become a new upstream version, too. Also good.

> So, if you agree, let's do some work together, and have a new
> version of Fizsh uploaded to Debian 8/Jessie. 

Will do.

> PS. I would not file a bug against the current Ubuntu package.

Seems no more necessary anyway. Against my suspection, this list
seemed to be a good place for this specific report.

But I wouldn't recommend doing that in general. If you are not sure
that it's an upstream issue, it's often better to report it to the
distribution where your zsh package came from first -- to not bother
upstream developers in case it's a distribution specific issue.

		Regards, Axel
-- 
 ,''`.  |  Axel Beckert <abe@debian.org>, http://people.debian.org/~abe/
: :' :  |  Debian Developer, ftp.ch.debian.org Admin
`. `'   |  1024D: F067 EA27 26B9 C3FC 1486  202E C09E 1D89 9593 0EDE
  `-    |  4096R: 2517 B724 C5F6 CA99 5329  6E61 2FF9 CD59 6126 16B5

Re: Zsh changes on Ubuntu

[Guido van Steen]

I looked into Thorsten's issue. I suspect it is caused by Fizsh being dependent on raw directional codes. Newer versions of Ubuntu seem to work with a different set of raw directional codes than older versions. For example, Ubuntu's raw code for "arrow-up" seems to have changed from "^[[A" to "^[0A". This would explain why things work on Debian and Cygwin but not on Ubuntu. 

I agree with Frank's suggestion to revert to more portable code like: 

> bindkey "${key[Up]}" 

This does not work on my Debian system (Stable distribution), but it can be made to work like this: 

> typeset -A key
> key[Up]=${terminfo[kcuu1]}

Guido 

Re: Zsh changes on Ubuntu

[Frank Terbeck]

Guido van Steen wrote:
> I looked into Thorsten's issue. I suspect it is caused by Fizsh being dependent
> on raw directional codes. Newer versions of Ubuntu seem to work with a
> different set of raw directional codes than older versions. For example,

Yes, that's very likely.

> Ubuntu's raw code for "arrow-up" seems to have changed from "^[[A" to "^[0A".

That's probably ^[OA, but that's one possible difference between the
different possible terminal modes.

> This would explain why things work on Debian and Cygwin but not on Ubuntu.
>
> I agree with Frank's suggestion to revert to more portable code like:
>
>> bindkey "${key[Up]}"
>
> This does not work on my Debian system (Stable distribution), but it
> can be made to work like this:
>
>> typeset -A key
>> key[Up]=${terminfo[kcuu1]}

Yes, that's what Debian's global zshrc is setting up.

However, like I said before, using the values from $terminfo[] is only
valid reliably if the terminal is in keyboard-transmit mode (via smkx).

Finding out if there's a hook in place, that sets that mode while zle is
in place is hard to do reliably.

Then there are other ways in zsh to do setup for special keys, like
‘zkbd’, that doesn't set the mode, but instead let's the user press all
sorts of special keys and records the escape sequences that produces.

‘zkbd’ also uses the $key[] map to store that information; and we used
the same keys in Debian's setup as ‘zkbd’ does.

So... Here's how I'd probably tackle the problem:

  - You want to bind the $foo key.
  - Check if $key[$foo] exists and is non-empty.
  - If so, use that value as the sequence to bind with.
  - If not, fall back to some hard-coded sequence, if you like.

That way you support setups that use Debian's way and zkbd, while
having a fallback in place. I think that's better than having a bunch
of half-assed tries to detect what situation you're in.


Regards, Frank

PS. Yay for terminals, right?

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Zsh changes on Ubuntu

[Guido van Steen]

For those interested in a new version of Fizsh: 

Version 1.0.6 has just been released at https://sourceforge.net/projects/fizsh/files/. Among other things the new version corrects the issue reported by Thorsten. A binary package for Debian and Ubuntu (and similar Linux distributions) has been provided as well.