Re: lexer issue

[Peter Stephenson <p.w.stephenson@ntlworld.com>]

On 28 March 2018 10:57:47 BST, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>Peter wrote:
>> Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
>> > Does anyone know what may be causing this:
>> > 
>> >     echo $(( ((##h << 8) + ##e) << 8)+<SPACE>
>> >   ??? echo $(( ((##h << 8) +
>> > 
>> > This is with space bound to magic-space.
>>
>> This will be to do with the fact that you've closed one of the
>> parentheses of the arithmetic expression, but not the other, so the
>next
>> expansion has decided it's a command substitution.  In that, the ##e
>is
>> a comment, so gets stripped.  In fact, I'm not seeing the effect you
>> are, which may be down to options, or to the fact that the way we
>handle
>> interactive comments changed recently, or a combination.

Sorry, limited network access at the moment so partial
poorly formatted reply.

This is indeed because it looks like a command subst that
went wrong. I think the math aspect is actually irrelevant.

>It can be reproduced starting from zsh -f with nothing more than
>  bindkey ' ' magic-space
>So interactive_comments is not set.

That appears to be important but I'm not sure why.

However, I think I do understand the basics of this.

When we  do the recursive parse of the command subst we
abort and flush the history on failure. But inside a string it
doesn't bother finishing off the history line. That's important in
this case because we're using that line directly as the string
we get from the nested lex.

Full patch when my laptop finds its way to Wi-Fi but for a taster:

In herrflush() , !strin should be (!strin || lex_add_raw) for which 
the static before lex_add_raw in lex. c has to go; inside the loop, in
addition to hwaddc() the return from ingetc() has also to be
passed to addtoline(). The !lexstop tests need tweaking as a consequence.

If that was too garbled I will be doing this properly next week. 

Haven't seen any side effects but they are endemic in the lexer /
ZLE interface so some playing around wil be necessary. 

pws