z parameter expansion flag is broken?

z parameter expansion flag is broken?

[Yuya Amemiya]

Why does z parameter expansion flag remove leading '-' from result?

in zsh 5.4.1

```
$ echo ${(z):--a}
a
$ echo ${(q):--a}
-a
```

in zsh 5.3.1

```
$ echo ${(z):--a}
-a
$ echo ${(q):--a}
-a
```

regards

--
Yuya Amemiya <ghostrevery@gmail.com>

Re: z parameter expansion flag is broken?

[Bart Schaefer]

On Sep 9,  4:13am, Yuya Amemiya wrote:
}
} Why does z parameter expansion flag remove leading '-' from result?

Hrm.

In the (z) case, "-" has been tokenized as DASH when bufferwords() is
called.  This eventually comes down to ingetc() while reading the next
lexical word from the input buffer.  At line 202 there is

202                 if (itok(lastc = STOUC(*inbufptr++)))
203                     continue;

Because itok(DASH) is true, we discard it and continue around the loop.

That code in ingetc() is as old as the shell itself, so I have to assume
that there's a good reason to discard token characters during lexing,
and the problem is upstream with the recently-introduced code that
more aggressively tokenizes hyphens ... though I don't understand why
other tokens haven't similarly been adversely affected (or maybe I just
haven't found how to get another token embedded in the string).

In any case this is a really serious bug; it's not just leading "-" that
get removed:

torch% print -r -- ${(z):--a -b -c d-e-f}
a b c def

Re: z parameter expansion flag is broken?

[Bart Schaefer]

On Sep 9,  9:53am, Bart Schaefer wrote:
}
} That code in ingetc() is as old as the shell itself, so I have to assume
} that there's a good reason to discard token characters during lexing,
} and the problem is upstream with the recently-introduced code that
} more aggressively tokenizes hyphens ... though I don't understand why
} other tokens haven't similarly been adversely affected (or maybe I just
} haven't found how to get another token embedded in the string).

Is there similarly something broken about the (Z+c+) flag?  Can somebody
show me an example where (Z+c+) and (Z+C+) produce different results?

(This is not new in zsh-5.4 though.)

Re: z parameter expansion flag is broken?

[Peter Stephenson]

On Sat, 9 Sep 2017 15:45:05 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> Is there similarly something broken about the (Z+c+) flag?  Can somebody
> show me an example where (Z+c+) and (Z+C+) produce different results?

See D04parameter.ztst, search for "(Z".

pws

Re: z parameter expansion flag is broken?

[Peter Stephenson]

On Sat, 09 Sep 2017 04:13:03 +0900 (JST)
Yuya Amemiya <ghostrevery@gmail.com> wrote:
> Why does z parameter expansion flag remove leading '-' from result?
> 
> in zsh 5.4.1
> 
> ```
> $ echo ${(z):--a}
> a

That's a bug, but the only new feature is that it now additionally affects
"-" because that's tokenized.  The basic effect is old:

% zsh-5.3.1 -f
% setopt nonomatch
% echo ${(z):-foo*bar}
foobar

Because "-" doesn't usually need quoting, it's more noticeable than with
other metacharacters.

Untokenizing before passing to bufferwords() is the easy fix, and the
result can only be better because anything altered would previously
simply have been dumped by the effect noted by Bart.  However, I'm not
at all convinced bufferwords() needs to dump token characters in this
case --- bufferwords() is a hack that originated as a helper for
completion but is now more widely used.  A more elegant solution (not on
the cards) wouldn't use bufferwords() at all.

pws

diff --git a/Src/subst.c b/Src/subst.c
index 5b1bf89..5df2a8b 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -3747,11 +3747,15 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags,
 
 	if (isarr) {
 	    char **ap;
-	    for (ap = aval; *ap; ap++)
+	    for (ap = aval; *ap; ap++) {
+		untokenize(*ap);
 		list = bufferwords(list, *ap, NULL, shsplit);
+	    }
 	    isarr = 0;
-	} else
+	} else {
+	    untokenize(val);
 	    list = bufferwords(NULL, val, NULL, shsplit);
+	}
 
 	if (!list || !firstnode(list))
 	    val = dupstring("");
diff --git a/Test/D04parameter.ztst b/Test/D04parameter.ztst
index 3c93990..8dbc1e8 100644
--- a/Test/D04parameter.ztst
+++ b/Test/D04parameter.ztst
@@ -2200,3 +2200,10 @@ F:behavior, see http://austingroupbugs.net/view.php?id=888
 >Option
 >Regular text
 >Option
+
+ (setopt nonomatch
+  print ${(z):-foo-bar*thingy?}
+ )
+0:(z) splitting with remaining tokens
+>foo-bar*thingy?
+