Various problems with 3.0.1-test1

Various problems with 3.0.1-test1

[Peter Stephenson]

Every time I investigate a problem, another turns up.  Here's the list
so far.

1) 

% ./zsh -fc 'shift 2; print $*' first second third fourth               
second third fourth fourth


2) This is very puzzling.

% ./zsh -fc 'setopt extendedglob;  [[ PATH = [A-Z]# ]] || print Failed.'
Failed.


3) This is minor but probably wrong:

% ./zsh -fc '1this_is_ignored=foo; print $1'
foo

ksh prints:
ksh: 1this_is_ignored=foo:  not found

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77330
Deutches Electronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

Re: Various problems with 3.0.1-test1

[Zoltan Hidvegi]

> 1) 
> 
> % ./zsh -fc 'shift 2; print $*' first second third fourth               
> second third fourth fourth

The fix is included below.  I broke it in 3.0.1-test.  It worked in 3.0.0
but in 3.0.0 shift array did not work when KSH_ARRAYS was set.

> 2) This is very puzzling.
> 
> % ./zsh -fc 'setopt extendedglob;  [[ PATH = [A-Z]# ]] || print Failed.'
> Failed.

This is know and probably not a bug.  extendedglob affects parsing and
parsing the -c argument is dode before anything is executed.  Similarily
aliases do not work here.

./zsh -o extendedglob -fc '[[ PATH = [A-Z]# ]] || print Failed.'

works.

> 3) This is minor but probably wrong:
> 
> % ./zsh -fc '1this_is_ignored=foo; print $1'
> foo
> 
> ksh prints:
> ksh: 1this_is_ignored=foo:  not found

It is probably trivial but I have no more time no (I would need some other
5 minutes for that).

Zoltan

*** Src/params.c	1996/09/07 15:19:40	2.37
--- Src/params.c	1996/09/10 13:49:02
***************
*** 964,972 ****
  {
      Value v;
  
!     if (!((v = getvalue(&s, 0)) && PM_TYPE(v->pm->flags) == PM_ARRAY))
! 	return NULL;
!     return v->pm->gets.afn(v->pm);
  }
  
  /**/
--- 964,973 ----
  {
      Value v;
  
!     if (!idigit(*s) && (v = getvalue(&s, 0)) &&
! 	PM_TYPE(v->pm->flags) == PM_ARRAY)
! 	return v->pm->gets.afn(v->pm);
!     return NULL;
  }
  
  /**/

Re: Various problems with 3.0.1-test1

[Peter Stephenson]

Zoltan Hidvegi wrote:
> > 2) This is very puzzling.
> > 
> > % ./zsh -fc 'setopt extendedglob;  [[ PATH = [A-Z]# ]] || print Failed.'
> > Failed.
> 
> This is know and probably not a bug.

I'm rather less sure.  The actual case that bothered me was more like
this.

% cat fntst
setopt extendedglob
[[ PATH = [A-Z]# ]] || print Failed
% ./zsh -fc 'fpath=.; autoload fntst; fntst'
Failed

This certainly looks to me like a bug.  I understand the point --- the
whole function is being parsed before extendedglob is set.  However,
there doesn't seem to be any way of making the function independent of
external options.  This is very unpleasant.  Explaining to people it's
not a bug because it's not supposed to work properly doesn't seem to
me the preferable answer.  It basically makes it impossible to use
extendedglob in portable functions.

In fact, the whole idea of the lexical tokens depending on some option
has me less than thrilled.  Since the only difference is in lextok2, I
think the following patch should work.  It has the bonus of making the
effect of extendedglob much more transparent in the code.  I would
urge its inclusion.  If you're counting, I've added seven extendedglob
tests and removed one.  I believe these are all the ones required.
(Maybe it was this way before?)

*** Src/glob.c.ext	Tue Sep 10 17:19:01 1996
--- Src/glob.c	Tue Sep 10 17:49:07 1996
***************
*** 788,800 ****
  		*str = '|';
  		break;
  	    } /* else fall through */
- 	case Pound:
- 	case Hat:
  	case Star:
  	case Inbrack:
  	case Inang:
  	case Quest:
  	    return str;
  	}
      if (!mod || parlev)
  	return NULL;
--- 788,802 ----
  		*str = '|';
  		break;
  	    } /* else fall through */
  	case Star:
  	case Inbrack:
  	case Inang:
  	case Quest:
  	    return str;
+ 	case Pound:
+ 	case Hat:
+ 	    if (isset(EXTENDEDGLOB))
+ 		return str;
  	}
      if (!mod || parlev)
  	return NULL;
***************
*** 1708,1715 ****
  	    pat++;
  	    continue;
  	}
! 	if (*pat == Hat)	/* following pattern is negated */
! 	    return 1 - doesmatch(c->next);
  	if (*pat == Inbrack) {
  	    /* Match groups of characters */
  #define PAT(X) (pat[X] == Meta ? pat[(X)+1] ^ 32 : untok(pat[X]))
--- 1710,1717 ----
  	    pat++;
  	    continue;
  	}
! 	if (*pat == Hat && isset(EXTENDEDGLOB))	
! 	    return 1 - doesmatch(c->next);  /* following pattern is negated */
  	if (*pat == Inbrack) {
  	    /* Match groups of characters */
  #define PAT(X) (pat[X] == Meta ? pat[(X)+1] ^ 32 : untok(pat[X]))
***************
*** 1897,1903 ****
  
      /* Parse repeated directories such as (dir/)# and (dir/)## */
      if (*(str = pptr) == Inpar && !skipparens(Inpar, Outpar, &str) &&
!         *str == Pound && str[-2] == '/') {
  	pptr++;
  	if (!(c1 = parsecompsw(0)))
  	    return NULL;
--- 1899,1905 ----
  
      /* Parse repeated directories such as (dir/)# and (dir/)## */
      if (*(str = pptr) == Inpar && !skipparens(Inpar, Outpar, &str) &&
!         *str == Pound && isset(EXTENDEDGLOB) && str[-2] == '/') {
  	pptr++;
  	if (!(c1 = parsecompsw(0)))
  	    return NULL;
***************
*** 1954,1960 ****
  	/* Go through code until we find something separating alternatives,
  	 * or path components if relevant.
  	 */
! 	if (*pptr == Hat) {
  	    /* negate remaining pattern */
  	    *s++ = Hat;
  	    *s++ = '\0';
--- 1956,1962 ----
  	/* Go through code until we find something separating alternatives,
  	 * or path components if relevant.
  	 */
! 	if (*pptr == Hat && isset(EXTENDEDGLOB)) {
  	    /* negate remaining pattern */
  	    *s++ = Hat;
  	    *s++ = '\0';
***************
*** 1995,2001 ****
  		errflag = 1;
  		return NULL;
  	    }
! 	    if (*pptr == Pound) {
  		/* Zero (or one) or more repetitions of group */
  		dpnd = 1;
  		pptr++;
--- 1997,2003 ----
  		errflag = 1;
  		return NULL;
  	    }
! 	    if (*pptr == Pound && isset(EXTENDEDGLOB)) {
  		/* Zero (or one) or more repetitions of group */
  		dpnd = 1;
  		pptr++;
***************
*** 2025,2031 ****
  	    c->str = dupstring(cstr);
  	    return c;
  	}
! 	if (*pptr == Pound) {
  	    /* repeat whatever we've just had (ls) zero or more times */
  	    *s = '\0';
  	    pptr++;
--- 2027,2033 ----
  	    c->str = dupstring(cstr);
  	    return c;
  	}
! 	if (*pptr == Pound && isset(EXTENDEDGLOB)) {
  	    /* repeat whatever we've just had (ls) zero or more times */
  	    *s = '\0';
  	    pptr++;
*** Src/lex.c.ext	Tue Sep 10 17:13:20 1996
--- Src/lex.c	Tue Sep 10 17:17:03 1996
***************
*** 315,320 ****
--- 315,322 ----
      lextok2['['] = Inbrack;
      lextok2['$'] = String;
      lextok2['~'] = Tilde;
+     lextok2['#'] = Pound;
+     lextok2['^'] = Hat;
  }
  
  /* initialize lexical state */
***************
*** 327,339 ****
      dbparens = alstat = lexstop = 0;
      incmdpos = 1;
      tok = ENDINPUT;
-     if (isset(EXTENDEDGLOB)) {
- 	lextok2['#'] = Pound;
- 	lextok2['^'] = Hat;
-     } else {
- 	lextok2['#'] = '#';
- 	lextok2['^'] = '^';
-     }
  }
  
  /* add a char to the string buffer */
--- 329,334 ----
*** Src/zle_tricky.c.ext	Tue Sep 10 17:18:53 1996
--- Src/zle_tricky.c	Tue Sep 10 17:25:21 1996
***************
*** 474,481 ****
  	    }
  	} else {
  	    /* Not a parameter expression so we check for wildcards */
! 	    if (*str == Pound || *str == Hat || *str == Star ||
! 		*str == Bar || *str == Quest ||
  		!skipparens(Inbrack, Outbrack, &str) ||
  		!skipparens(Inang,   Outang,   &str) ||
  		(unset(IGNOREBRACES) &&
--- 474,481 ----
  	    }
  	} else {
  	    /* Not a parameter expression so we check for wildcards */
! 	    if (((*str == Pound || *str == Hat) && isset(EXTENDEDGLOB)) ||
! 		*str == Star || *str == Bar || *str == Quest ||
  		!skipparens(Inbrack, Outbrack, &str) ||
  		!skipparens(Inang,   Outang,   &str) ||
  		(unset(IGNOREBRACES) &&


-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77330
Deutches Electronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

Re: Various problems with 3.0.1-test1

[Zoltan Hidvegi]

> In fact, the whole idea of the lexical tokens depending on some option
> has me less than thrilled.  Since the only difference is in lextok2, I
> think the following patch should work.  It has the bonus of making the
> effect of extendedglob much more transparent in the code.  I would
> urge its inclusion.  If you're counting, I've added seven extendedglob
> tests and removed one.  I believe these are all the ones required.
> (Maybe it was this way before?)

No I think zsh-2.5.0 had the same behaviour.  Your patches to lex.c and
zle_tricky.c are fine but for glob.c the second hunk is unnecessary and
undesired.  The second hunk changes the glob engine.  Here Hat comes from
the pattern parser and denotes negation.  It purely represents the concept
of negation and it is not related to the actual ^ mark).  Later when we
implement ksh globbing only the pattern parser will be changed and in the
parsed Comp structure Hat will still denote !(pattern) negation.

And a change is necessary in subst.c as well because after your patch
$foo:s/^/something/ stopps working.  Perhaps it was a bug even before that
this never worked with extendedglob set.

We also have to double-check all places where the tokenized lexer output is
used since an unexpected Hat or Pound token might cause problems.
For example in the result of get_comp_string() although right now I'm
unable to produce any bug related to this.

Zoltan


*** Src/subst.c	1996/07/28 22:29:59	2.48
--- Src/subst.c	1996/09/10 23:30:38
***************
*** 1568,1573 ****
--- 1568,1574 ----
  		for (tt = hsubl; *tt; tt++)
  		    if (INULL(*tt))
  			chuck(tt--);
+ 		untokenize(hsubl);
  		for (tt = hsubr = ztrdup(ptr2); *tt; tt++)
  		    if (INULL(*tt))
  			chuck(tt--);
*** Src/glob.c.orig	Wed Sep 11 00:51:32 1996
--- Src/glob.c	Wed Sep 11 01:33:59 1996
***************
*** 1710,1717 ****
  	    pat++;
  	    continue;
  	}
! 	if (*pat == Hat && isset(EXTENDEDGLOB))	
! 	    return 1 - doesmatch(c->next);  /* following pattern is negated */
  	if (*pat == Inbrack) {
  	    /* Match groups of characters */
  #define PAT(X) (pat[X] == Meta ? pat[(X)+1] ^ 32 : untok(pat[X]))
--- 1710,1717 ----
  	    pat++;
  	    continue;
  	}
! 	if (*pat == Hat)	/* following pattern is negated */
! 	    return 1 - doesmatch(c->next);
  	if (*pat == Inbrack) {
  	    /* Match groups of characters */
  #define PAT(X) (pat[X] == Meta ? pat[(X)+1] ^ 32 : untok(pat[X]))