[BUG] (z)-flag interrupts if $() in input

[BUG] (z)-flag interrupts if $() in input

[Sebastian Gniazdowski]

Hello,
% a='This is a $() test of z flag'
% print -rl -- "${(z@)a}"
This
is
a
$()

% a='This is a $(a) test of z flag'
% print -rl -- "${(z@)a}"
This
is
a
$(a)
test
of
z
flag

-- 
  Sebastian Gniazdowski
  psprint@fastmail.com

Re: [BUG] (z)-flag interrupts if $() in input

[Daniel Shahaf]

Sebastian Gniazdowski wrote on Sat, Oct 29, 2016 at 23:33:09 -0700:
> Hello,
> % a='This is a $() test of z flag'
> % print -rl -- "${(z@)a}"
> This
> is
> a
> $()
> 

«eval $a» returns a parse error in zsh but succeeds in sh.  Let's put
(z) to one side for the moment; should the command
.
    % echo foo $() bar
.
print "foo bar\n" or return a parse error?

Re: [BUG] (z)-flag interrupts if $() in input

[Peter Stephenson]

On Sun, 30 Oct 2016 15:25:25 +0000
Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Should the command
> .
>     % echo foo $() bar
> .
> print "foo bar\n" or return a parse error?
> 

That's obviously a bug.  The fix is easy, although I don't know why the
case of end of input in a normal input sequence, just above, needs to be
different.

pws

diff --git a/Src/parse.c b/Src/parse.c
index 24b8cd9..50a0d5f 100644
--- a/Src/parse.c
+++ b/Src/parse.c
@@ -594,7 +594,7 @@ par_event(int endtok)
     if (tok == ENDINPUT)
 	return 0;
     if (tok == endtok)
-	return 0;
+	return 1;
 
     p = ecadd(0);
 
diff --git a/Test/D08cmdsubst.ztst b/Test/D08cmdsubst.ztst
index 89e7259..3625373 100644
--- a/Test/D08cmdsubst.ztst
+++ b/Test/D08cmdsubst.ztst
@@ -153,3 +153,17 @@
   eval 'foo echo this just works, OK\?)'
 0:backtracking within command string parsing with alias still pending
 >this just works, OK?
+
+  (
+    set errexit
+    show_nargs() { print $#; }
+    print a $() b
+    print c "$()" d
+  )
+0:Empty $() is a valid empty substitution.
+>a b
+>c  d
+
+  empty=$() && print "'$empty'"
+0:Empty $() is a valid assignment
+>''

Re: [BUG] (z)-flag interrupts if $() in input

[Bart Schaefer]

On Oct 30,  3:25pm, Daniel Shahaf wrote:
}
} "eval $a" returns a parse error in zsh but succeeds in sh.  Let's put
} (z) to one side for the moment;

This actually is a question about (z) rather than about parse errors,
because other parse errors don't have the same problem:

torch% echo ||| print
zsh: parse error near `|'
torch% a='echo ||| print'
torch% print -rl -- ${(z)a}
echo
||
|
print

So the question is not whether $() is a parse error, but why $() eats
the rest of the command line.

Which I think is going to come down to the way cmd_or_math() has to
read ahead to discover whether $( starts a math expresion or a
command substitution -- but I haven't dug in that far.

Re: [BUG] (z)-flag interrupts if $() in input

[Bart Schaefer]

On Oct 30,  5:09pm, Peter Stephenson wrote:
}
} Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
} > Should the command
} > .
} >     % echo foo $() bar
} > .
} > print "foo bar\n" or return a parse error?
} 
} That's obviously a bug.  The fix is easy, although I don't know why the
} case of end of input in a normal input sequence, just above, needs to be
} different.

This solves the specific example but doesn't solve the general problem;
if there is a parse error inside $(...) then ${(z)...} fails.

torch% a='x $() y'
torch% print -rl -- ${(z)a}
x
$()
y
torch% a='x $(|||) y'
torch% print -rl -- ${(z)a}
x
$(||
torch% 

Re: [BUG] (z)-flag interrupts if $() in input

[Bart Schaefer]

On Oct 30, 11:04am, Bart Schaefer wrote:
}
} torch% a='x $(|||) y'
} torch% print -rl -- ${(z)a}
} x
} $(||
} torch% 

A little progress but not much.

If we look at this:

torch% echo x $(|||) y
zsh: parse error near `||'
zsh: parse error near `$(|||) y'

Note that we have one parse error inside $(...) and then another for
the whole expression.

It's the inner parse error that kills (z).  gettokstr() returns LEXERR
to bufferwords() and it stops building the array.  But ... even if I
force the token to STRING there to make the loop continue, the next
ctxlex() returns ENDINPUT with empty tokstr, so the result isn't changed.

Re: [BUG] (z)-flag interrupts if $() in input

[Peter Stephenson]

On Sun, 30 Oct 2016 11:04:55 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> This solves the specific example but doesn't solve the general problem;
> if there is a parse error inside $(...) then ${(z)...} fails.
> 
> torch% a='x $() y'
> torch% print -rl -- ${(z)a}
> x
> $()
> y
> torch% a='x $(|||) y'
> torch% print -rl -- ${(z)a}
> x
> $(||

This fixes it up in the usual way, i.e. read to the end of the string
but don't attempt to parse further.  It's easy since we're still in the
mode where the command substitution is being copied to the token on
input, so just carry on doing that until the end of the string.

pws

diff --git a/Src/lex.c b/Src/lex.c
index e0935bf..8896128 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -2138,8 +2138,17 @@ skipcomm(void)
     lexflags &= ~LEXFLAGS_ZLE;
     dbparens = 0;	/* restored by zcontext_restore_partial() */
 
-    if (!parse_event(OUTPAR) || tok != OUTPAR)
-	lexstop = 1;
+    if (!parse_event(OUTPAR) || tok != OUTPAR) {
+	if (strin) {
+	    /*
+	     * Get the rest of the string raw since we don't
+	     * know where this token ends.
+	     */
+	    while (!lexstop)
+		(void)ingetc();
+	} else
+	    lexstop = 1;
+    }
      /* Outpar lexical token gets added in caller if present */
 
     /*
diff --git a/Test/D04parameter.ztst b/Test/D04parameter.ztst
index 7623051..97c8ba3 100644
--- a/Test/D04parameter.ztst
+++ b/Test/D04parameter.ztst
@@ -631,6 +631,14 @@
 >;
 >(( echo 42 
 
+  # From parse error on it's not possible to split.
+  # Just check we get the complete string.
+  foo='echo $(|||) bar'
+  print -rl ${(z)foo}
+0:$($(z)} with parse error in command substitution.
+>echo
+>$(|||) bar
+
   psvar=(dog)
   setopt promptsubst
   foo='It shouldn'\''t $(happen) to a %1v.'