Fwd: [Pkg-zsh-devel] Bug#793168: zsh: expand-substitution-on-TAB broke for $(()) in 5.0.8 [origin: adi@hexapodia.org]

Fwd: [Pkg-zsh-devel] Bug#793168: zsh: expand-substitution-on-TAB broke for $(()) in 5.0.8 [origin: adi@hexapodia.org]

[Axel Beckert]

Hi,

this bug report has been filed against zsh 5.0.8 in Debian:

  https://bugs.debian.org/793168

I've skimmed over the commits and zsh-workers mails since the release
and didn't notice anything covering this issue yet.

I can also reproduce this issue with zsh 5.0.8 in Debian and can
confirm, that it didn't happen with zsh 5.0.7 in Debian.

----- Forwarded message from Andy Isaacson <adi@hexapodia.org> -----
Date: Tue, 21 Jul 2015 16:00:43 -0700
From: Andy Isaacson <adi@hexapodia.org>
Subject: [Pkg-zsh-devel] Bug#793168: zsh: expand-substitution-on-TAB broke for $(()) in 5.0.8
Reply-To: Andy Isaacson <adi@hexapodia.org>, 793168@bugs.debian.org

Package: zsh
Version: 5.0.8-3
Severity: normal

Dear Maintainer,

with zsh 5.0.7 and earlier versions, if I type $((5*8)) and then hit TAB,
the expression is replaced with its evaluation ("40" in this case). The
same feature works with many different substitutions.

as of 5.0.8 TAB-substituting does not work with $(()) anymore.

It does still work right with "${VAR}" and "$[5+8]".

the substitution does get done before running the command:

t440s% echo $((5*8))
40

seems like setopt might influence this behavior:

t440s% setopt
nobeep
globdots
interactive
login
monitor
shinstdin
zle


-- Package-specific info:

Packages which provide vendor completions:

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                        Version            Architecture       Description
+++-===========================-==================-==================-============================================================
ii  pulseaudio                  6.0-2              amd64              PulseAudio sound server
ii  systemd                     222-2              amd64              system and service manager
ii  udev                        222-2              amd64              /dev/ and hotplug management daemon
ii  vlc-nox                     2.2.1-2+b1         amd64              multimedia player and streamer (without X support)

dpkg-query: no path found matching pattern /usr/share/zsh/vendor-functions/


-- System Information:
Debian Release: stretch/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.1.2 (SMP w/4 CPU cores)
Locale: LANG=en_US.utf8, LC_CTYPE=en_US.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages zsh depends on:
ii  dpkg        1.18.1
ii  libc6       2.19-19
ii  libcap2     1:2.24-9
ii  libtinfo5   5.9+20150516-2
ii  zsh-common  5.0.8-3

Versions of packages zsh recommends:
ii  libncursesw5  5.9+20150516-2
ii  libpcre3      2:8.35-7

Versions of packages zsh suggests:
ii  zsh-doc  5.0.8-3

-- no debconf information
----- End forwarded message -----

		Regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert, PGP: 612616B5
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://arc.pasp.de/                      | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: Fwd: [Pkg-zsh-devel] Bug#793168: zsh: expand-substitution-on-TAB broke for $(()) in 5.0.8 [origin: adi@hexapodia.org]

[Bart Schaefer]

On Jul 25,  7:03pm, Axel Beckert wrote:
}
} I can also reproduce this issue with zsh 5.0.8 in Debian and can
} confirm, that it didn't happen with zsh 5.0.7 in Debian.
} 
} ----- Forwarded message from Andy Isaacson <adi@hexapodia.org> -----
} 
} with zsh 5.0.7 and earlier versions, if I type $((5*8)) and then hit TAB,
} the expression is replaced with its evaluation ("40" in this case).

This appears to be a difference in behavior of the expand-or-complete
built-in widget.  With _expand added to the completer zstyle or with
the expand-word widget, the behavior is unchanged.

There's no change to the widget itself.  The issue seems to be that
the completion system is examining $((5*8) [without the trailing paren] 
insead of the full expression.  You can see this in action if you
"setopt completealiases"; using "_" to represent the position of the
cursor:

torch% setopt completealiases
torch% echo $((5*8))_<TAB>
[... list of parameter names appears ...]
torch% echo $((5*8)_)

This must in turn be due to the change in the parser to disambiguate
"$((" as either math or command substitution; completion is trying it
as command substitution without looking ahead to see if it can be done
as math instead.

Re: [Pkg-zsh-devel] Bug#793168: zsh: expand-substitution-on-TAB broke for $(()) in 5.0.8 [origin: adi@hexapodia.org]

[Peter Stephenson]

On Sat, 25 Jul 2015 19:03:41 +0200
Axel Beckert <abe@deuxchevaux.org> wrote:
> with zsh 5.0.7 and earlier versions, if I type $((5*8)) and then hit TAB,
> the expression is replaced with its evaluation ("40" in this case). The
> same feature works with many different substitutions.
> 
> as of 5.0.8 TAB-substituting does not work with $(()) anymore.
> 
> It does still work right with "${VAR}" and "$[5+8]".

There are two things here: the _expand completer and the
expand-or-complete widget.  The expand-word widget still works.

I think the patch below fixes the expand-or-complete case.

The _expand case is in shell code.  The difference apparently
comes from the fact that in the new code $(( ... )) gets
tokenised such that the outer parentheses become Inparmath
and Outparmath, while the inner parentheses don't get tokenised.
That means that at line 25 of _expand,

  word="$IPREFIX$PREFIX$SUFFIX$ISUFFIX"

becomes (from ^X?)

 +_expand:25> word='$(\(3\*4\))' 

so that the expansion thinks the inner parentheses should be treated
literally.  (I think $PREFIX is the only relevant word in the normal
case, but I think the same rules apply to all the variables.)

One fix might be to attempt to replace (\(...\)) globally with
((...)).  That might be OK in most cases, but as we're doing it in shell
code it's not perfect syntactically.

It might be possible to detect this at the point where we introduce the
backslashes, but I don't actually know where that is.  Does anyone?

I'll think about one of these anyone else can see a better alternative
(pontificating about the possbility of a better alternative doesn't
count :-)).

It *might* be OK to tokenise the inner parentheses in lex.c, but I don't
really want to tinker with the main shell unless I have to.

pws

diff --git a/Src/Zle/zle_tricky.c b/Src/Zle/zle_tricky.c
index 2104ca1..3bf8d45 100644
--- a/Src/Zle/zle_tricky.c
+++ b/Src/Zle/zle_tricky.c
@@ -721,11 +721,12 @@ docomplete(int lst)
 		    }
 		}
 	    }
-	    if (lst == COMP_EXPAND_COMPLETE)
+	    if (lst == COMP_EXPAND_COMPLETE) {
 		do {
 		    /* Check if there is a parameter expression. */
 		    for (; *q && *q != String; q++);
-		    if (*q == String && q[1] != Inpar && q[1] != Inbrack) {
+		    if (*q == String && q[1] != Inpar && q[1] != Inparmath &&
+			q[1] != Inbrack) {
 			if (*++q == Inbrace) {
 			    if (! skipparens(Inbrace, Outbrace, &q) &&
 				q == s + zlemetacs - wb)
@@ -769,6 +770,7 @@ docomplete(int lst)
 		    } else
 			break;
 		} while (q < s + zlemetacs - wb);
+	    }
 	    if (lst == COMP_EXPAND_COMPLETE) {
 		/* If it is still not clear if we should use expansion or   *
 		 * completion and there is a `$' or a backtick in the word, *

Re: [Pkg-zsh-devel] Bug#793168: zsh: expand-substitution-on-TAB broke for $(()) in 5.0.8 [origin: adi@hexapodia.org]

[Peter Stephenson]

On Sat, 25 Jul 2015 20:15:09 +0100
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> On Sat, 25 Jul 2015 19:03:41 +0200
> Axel Beckert <abe@deuxchevaux.org> wrote:
> > with zsh 5.0.7 and earlier versions, if I type $((5*8)) and then hit TAB,
> > the expression is replaced with its evaluation ("40" in this case). The
> > same feature works with many different substitutions.
> > 
> > as of 5.0.8 TAB-substituting does not work with $(()) anymore.
> > 
> > It does still work right with "${VAR}" and "$[5+8]".
> I think the patch below fixes the expand-or-complete case.
> 
> The _expand case is in shell code.  The difference apparently
> comes from the fact that in the new code $(( ... )) gets
> tokenised such that the outer parentheses become Inparmath
> and Outparmath, while the inner parentheses don't get tokenised.

This fixes the standard quoting code, which is used by completion in a
slightly non-standard (surprised?) way, with tokens still present
in the string, so we can detect the math expression and avoid quoting it
further.  _expand now works.

This is evidently not how $[...] gets handled, but as we have syntactic
markers in the present case we might as well make use of them.

pws

diff --git a/Src/utils.c b/Src/utils.c
index 0acab88..f7aaaed 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -5551,7 +5551,25 @@ quotestring(const char *s, char **e, int instring)
 		/* Needs to be passed straight through. */
 		if (dobackslash)
 		    *v++ = '\\';
-		*v++ = *u++;
+		if (*u == Inparmath) {
+		    /*
+		     * Already syntactically quoted: don't
+		     * add more.
+		     */
+		    int inmath = 1;
+		    *v++ = *u++;
+		    for (;;) {
+			char uc = *u;
+			*v++ = *u++;
+			if (uc == '\0')
+			    break;
+			else if (uc == Outparmath && !--inmath)
+			    break;
+			else if (uc == Inparmath)
+			    ++inmath;
+		    }
+		} else
+		    *v++ = *u++;
 		continue;
 	    }