Bug in ${(z)...} lexing, or what?

Bug in ${(z)...} lexing, or what?

[Bart Schaefer]

This has me baffled.

Consider the following innocuous-looking function:

widget() {
   zle up-history
   lastcmd=( ${(z)BUFFER} )
   zle down-history
}

Now try this:

zagzig% zle -N widget
zagzig% bindkey '\e=' widget
zagzig% ( setopt extendedglob ; [[ 'foo[x]=y' = (#i)[[:alpha:]][][[:alnum:]]#=* ]] )
zagzig% echo $lastcmd
( setopt extendedglob ; [[ 'foo[x]=y' = ( ;
zagzig% 

What happened to everything after the '#' in '(#i)' ??  And where did that
extra semicolon come from?

The really strange thing is that this happens only in ZLE widgets, and only
when there's a command separator (semi, amper, pipe, etc.) somewhere to the
left of the [[ ]] expression.  If I try the same thing outside of ZLE, it
works perfectly; if I try a bare glob (not in [[ ... ]]) it also works.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Bug in ${(z)...} lexing, or what?

[Sven Wischnowsky]

Bart Schaefer wrote:

> This has me baffled.
> 
> Consider the following innocuous-looking function:
> 
> widget() {
>    zle up-history
>    lastcmd=( ${(z)BUFFER} )
>    zle down-history
> }
> 
> Now try this:
> 
> zagzig% zle -N widget
> zagzig% bindkey '\e=' widget
> zagzig% ( setopt extendedglob ; [[ 'foo[x]=y' = (#i)[[:alpha:]][][[:alnum:]]#=* ]] )
> zagzig% echo $lastcmd
> ( setopt extendedglob ; [[ 'foo[x]=y' = ( ;
> zagzig% 
> 
> What happened to everything after the '#' in '(#i)' ??  And where did that
> extra semicolon come from?
> 
> The really strange thing is that this happens only in ZLE widgets, and only
> when there's a command separator (semi, amper, pipe, etc.) somewhere to the
> left of the [[ ]] expression.  If I try the same thing outside of ZLE, it
> works perfectly; if I try a bare glob (not in [[ ... ]]) it also works.

You have interactive_comments set, right?

This should fix that (I haven't looked close enough to find out when
it did that and when it didn't).

There is still something fishy, though. A string like `ls (#i)foo' is
broken into `ls' and `(#i)foo', but `[[ (#i)foo ]]' is broken into
`[[', `(', `#i', `)', ... I haven't found out where and why this
happens yet.


Bye
 Sven

Index: Src/hist.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/hist.c,v
retrieving revision 1.8
diff -u -r1.8 hist.c
--- Src/hist.c	2000/05/30 14:08:48	1.8
+++ Src/hist.c	2000/07/12 10:29:12
@@ -2058,15 +2058,15 @@
 bufferwords(LinkList list, char *buf, int *index)
 {
     int num = 0, cur = -1, got = 0, ne = noerrs, ocs = cs;
-    int owb = wb, owe = we, oadx = addedx, ozp = zleparse;
+    int owb = wb, owe = we, oadx = addedx, ozp = zleparse, oexp = expanding;
     char *p;
 
     if (!list)
 	list = newlinklist();
 
-    zleparse = 1;
+    zleparse = 3;
     addedx = 0;
-    noerrs = 1;
+    noerrs = expanding = 1;
     lexsave();
     if (buf) {
 	int l = strlen(buf);
@@ -2133,6 +2133,7 @@
     inpop();
     errflag = 0;
     zleparse = ozp;
+    expanding = oexp;
     noerrs = ne;
     lexrestore();
     cs = ocs;
Index: Src/lex.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/lex.c,v
retrieving revision 1.8
diff -u -r1.8 lex.c
--- Src/lex.c	2000/06/06 08:35:54	1.8
+++ Src/lex.c	2000/07/12 10:29:12
@@ -673,7 +673,7 @@
     /* chars in initial position in word */
 
     if (c == hashchar &&
-	(isset(INTERACTIVECOMMENTS) ||
+	((zleparse != 3 && isset(INTERACTIVECOMMENTS)) ||
 	 (!zleparse && !expanding &&
 	  (!interact || unset(SHINSTDIN) || strin)))) {
 	/* History is handled here to prevent extra  *

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Bug in ${(z)...} lexing, or what?

[Peter Stephenson]

Sven wrote:
> There is still something fishy, though. A string like `ls (#i)foo' is
> broken into `ls' and `(#i)foo', but `[[ (#i)foo ]]' is broken into
> `[[', `(', `#i', `)', ... I haven't found out where and why this
> happens yet.

I can partly answer that.  There's a context dependency in conditions,
because a `(' may introduce the start of a pattern, as here, or it may
introduce the start of a group.  Luckily, the former only happens when we
are expecting an argument to a test (you can't have patterns on the left of
an `=' or `!=', otherwise the issue would have been unresolvable) and the
latter when we are expecting a complete test, so we can check.  In your
example, `(' should indeed be a single token introducing a group, so the
parsing is correct.  I don't know exactly what happens on the right of an
`=', but it's possible that in that case, too, the `(' is lexed before we
decide and the string put together later, but it may also be normal.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: Bug in ${(z)...} lexing, or what?

[Bart Schaefer]

On Jul 12, 12:29pm, Sven Wischnowsky wrote:
} Subject: Re: Bug in ${(z)...} lexing, or what?
}
} Bart Schaefer wrote:
} 
} > This has me baffled.
} 
} You have interactive_comments set, right?

No, I *don't*, which contributed significantly to my confusion.

} This should fix that (I haven't looked close enough to find out when
} it did that and when it didn't).

This seems to have fixed it whether or not interactive_comments is set;
does at least one of us understand why?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Bug in ${(z)...} lexing, or what?

[Sven Wischnowsky]

Peter Stephenson wrote:

> Sven wrote:
> > There is still something fishy, though. A string like `ls (#i)foo' is
> > broken into `ls' and `(#i)foo', but `[[ (#i)foo ]]' is broken into
> > `[[', `(', `#i', `)', ... I haven't found out where and why this
> > happens yet.
> 
> I can partly answer that.  There's a context dependency in conditions,
> because a `(' may introduce the start of a pattern, as here, or it may
> introduce the start of a group.  Luckily, the former only happens when we
> are expecting an argument to a test (you can't have patterns on the left of
> an `=' or `!=', otherwise the issue would have been unresolvable) and the
> latter when we are expecting a complete test, so we can check.  In your
> example, `(' should indeed be a single token introducing a group, so the
> parsing is correct.  I don't know exactly what happens on the right of an
> `=', but it's possible that in that case, too, the `(' is lexed before we
> decide and the string put together later, but it may also be normal.

I should have used this example:

  % a='[[ a = (#i)foo ]]'
  % print -l ${(z)a}
  [[
  a
  =
  (
  #i
  )
  foo
  ]]


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: Bug in ${(z)...} lexing, or what?

[Peter Stephenson]

>   % a='[[ a = (#i)foo ]]'
>   % print -l ${(z)a}
>   [[
>   a
>   =
>   (
>   #i
>   )
>   foo
>   ]]

Strange, I get (#i)foo on one line, even with zsh -f.  There must be some
unexpected option dependence, but I can't see what.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: Bug in ${(z)...} lexing, or what?

[Peter Stephenson]

> >   % a='[[ a = (#i)foo ]]'
> >   % print -l ${(z)a}
> >   [[
> >   a
> >   =
> >   (
> >   #i
> >   )
> >   foo
> >   ]]
> 
> Strange, I get (#i)foo on one line, even with zsh -f.  There must be some
> unexpected option dependence, but I can't see what.

That was before applying the patch (the correct behaviour).  After, I get:

[[
a
=
(
;

with interactivecomments set (which is wrong), and what Sven was reporting
without it (which isn't great).  I'd much prefer the old behaviour in this
case, but the interactivecomments variant is definitely broken.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: Bug in ${(z)...} lexing, or what?

[Bart Schaefer]

On Jul 12,  5:57pm, Peter Stephenson wrote:
} Subject: Re: Bug in ${(z)...} lexing, or what?
}
} That was before applying the patch (the correct behaviour).  After, I get:
} 
} [[
} a
} =
} (
} ;
} 
} with interactivecomments set (which is wrong), and what Sven was reporting
} without it (which isn't great).  I'd much prefer the old behaviour in this
} case, but the interactivecomments variant is definitely broken.

The situation is exactly reversed inside a ZLE widget:  The behavior with
both interactivecomments and without is now correct inside the widget, but
broken outside.

So was all Sven's patch acheived to invert some condition?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net