lexer issue

lexer issue

[Oliver Kiddle]

Does anyone know what may be causing this:

    echo $(( ((##h << 8) + ##e) << 8)+<SPACE>
  → echo $(( ((##h << 8) +

This is with space bound to magic-space.

Also notable is that if you press <ENTER> there are TWO error messages:

zsh: parse error near `+'
zsh: parse error near `$(( ((##h << 8) + ##...'

I've not had much success in trying to understand this. The magic-space
expansion takes place in doexpandhist() but I think it has something
more to do with the errflag values and the parser, neither of which I
have much understanding of.

Oliver

Re: lexer issue

[Peter Stephenson]

On Tue, 27 Mar 2018 13:05:38 +0200
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Does anyone know what may be causing this:
> 
>     echo $(( ((##h << 8) + ##e) << 8)+<SPACE>
>   → echo $(( ((##h << 8) +
> 
> This is with space bound to magic-space.

This will be to do with the fact that you've closed one of the
parentheses of the arithmetic expression, but not the other, so the next
expansion has decided it's a command substitution.  In that, the ##e is
a comment, so gets stripped.  In fact, I'm not seeing the effect you
are, which may be down to options, or to the fact that the way we handle
interactive comments changed recently, or a combination.

pws

Re: lexer issue

[Oliver Kiddle]

Peter wrote:
> Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> > Does anyone know what may be causing this:
> > 
> >     echo $(( ((##h << 8) + ##e) << 8)+<SPACE>
> >   ??? echo $(( ((##h << 8) +
> > 
> > This is with space bound to magic-space.
>
> This will be to do with the fact that you've closed one of the
> parentheses of the arithmetic expression, but not the other, so the next
> expansion has decided it's a command substitution.  In that, the ##e is
> a comment, so gets stripped.  In fact, I'm not seeing the effect you
> are, which may be down to options, or to the fact that the way we handle
> interactive comments changed recently, or a combination.

It can be reproduced starting from zsh -f with nothing more than
  bindkey ' ' magic-space
So interactive_comments is not set.

I managed to bisect it and it was introduced with 34160 (c0d01a6).

Whether it is parsed as a command substitution or arithmetic expression,
there's nothing in there that might be interpreted as being a history
expansion so magic-space should not do anything regardless of how the
parser sees it.

Oliver

Re: lexer issue

[Peter Stephenson]

On 28 March 2018 10:57:47 BST, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>Peter wrote:
>> Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>> > Does anyone know what may be causing this:
>> > 
>> >     echo $(( ((##h << 8) + ##e) << 8)+<SPACE>
>> >   ??? echo $(( ((##h << 8) +
>> > 
>> > This is with space bound to magic-space.
>>
>> This will be to do with the fact that you've closed one of the
>> parentheses of the arithmetic expression, but not the other, so the
>next
>> expansion has decided it's a command substitution.  In that, the ##e
>is
>> a comment, so gets stripped.  In fact, I'm not seeing the effect you
>> are, which may be down to options, or to the fact that the way we
>handle
>> interactive comments changed recently, or a combination.

Sorry, limited network access at the moment so partial
poorly formatted reply.

This is indeed because it looks like a command subst that
went wrong. I think the math aspect is actually irrelevant.

>It can be reproduced starting from zsh -f with nothing more than
>  bindkey ' ' magic-space
>So interactive_comments is not set.

That appears to be important but I'm not sure why.

However, I think I do understand the basics of this.

When we  do the recursive parse of the command subst we
abort and flush the history on failure. But inside a string it
doesn't bother finishing off the history line. That's important in
this case because we're using that line directly as the string
we get from the nested lex.

Full patch when my laptop finds its way to Wi-Fi but for a taster:

In herrflush() , !strin should be (!strin || lex_add_raw) for which 
the static before lex_add_raw in lex. c has to go; inside the loop, in
addition to hwaddc() the return from ingetc() has also to be
passed to addtoline(). The !lexstop tests need tweaking as a consequence.

If that was too garbled I will be doing this properly next week. 

Haven't seen any side effects but they are endemic in the lexer /
ZLE interface so some playing around wil be necessary. 

pws

Re: lexer issue

[Peter Stephenson]

On Sat, 31 Mar 2018 18:17:21 +0100
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> When we  do the recursive parse of the command subst we
> abort and flush the history on failure. But inside a string it
> doesn't bother finishing off the history line. That's important in
> this case because we're using that line directly as the string
> we get from the nested lex.

Here's the patch.  Second test release will follow.

pws

diff --git a/Src/hist.c b/Src/hist.c
index b798be8..dbdc1e4 100644
--- a/Src/hist.c
+++ b/Src/hist.c
@@ -465,8 +465,26 @@ herrflush(void)
 {
     inpopalias();
 
-    while (!lexstop && inbufct && !strin)
-	hwaddc(ingetc());
+    if (lexstop)
+	return;
+    /*
+     * The lex_add_raw test is needed if we are parsing a command
+     * substitution when expanding history for ZLE: strin is set but we
+     * need to finish off the input because the string we are reading is
+     * going to be used directly in the line that goes to ZLE.
+     *
+     * Note that this is a side effect --- this is not the usual reason
+     * for testing lex_add_raw which is to add the text to a different
+     * buffer used when we are actually parsing the command substituion
+     * (nothing to do with ZLE).  Sorry.
+     */
+    while (inbufct && (!strin || lex_add_raw)) {
+	int c = ingetc();
+	if (!lexstop) {
+	    hwaddc(c);
+	    addtoline(c);
+	}
+    }
 }
 
 /*
diff --git a/Src/lex.c b/Src/lex.c
index 2379804..44ad880 100644
--- a/Src/lex.c
+++ b/Src/lex.c
@@ -158,7 +158,7 @@ mod_export int nocomments;
 /* add raw input characters while parsing command substitution */
 
 /**/
-static int lex_add_raw;
+int lex_add_raw;
 
 /* variables associated with the above */
 
diff --git a/Test/X03zlebindkey.ztst b/Test/X03zlebindkey.ztst
index 013d3df..298d7df 100644
--- a/Test/X03zlebindkey.ztst
+++ b/Test/X03zlebindkey.ztst
@@ -126,3 +126,20 @@
 >CURSOR: 1
 >BUFFER: ホ
 >CURSOR: 1
+
+  zpty_run 'bindkey " " magic-space'
+  setopt interactivecomments
+  zletest 'echo $(( x ) x ) y'
+  zletest 'echo $(( ##x ) ##x ) y'
+  unsetopt interactivecomments
+  zletest 'echo $(( x ) x ) y'
+  zletest 'echo $(( ##x ) ##x ) y'
+0:history expansion of failed command substitution using magic-space binding
+>BUFFER: echo $(( x ) x ) y
+>CURSOR: 18
+>BUFFER: echo $(( ##x ) ##x ) y
+>CURSOR: 22
+>BUFFER: echo $(( x ) x ) y
+>CURSOR: 18
+>BUFFER: echo $(( ##x ) ##x ) y
+>CURSOR: 22