Parameter substitution bug

Parameter substitution bug

[Peter Stephenson]

Can anybody explain this?

% cat tst
print $HOME/bin
print ~/bin
fdir='~/bin'
print ${~fdir}
setopt globsubst
print $fdir
% zsh -f ./tst
/home/user2/pws/bin
/home/user2/pws/bin
~/bin
~/bin

Why don't the last two give the same as the first two?  There's no rule
that says `parameter expansion doesn't work properly in non-interactive
shells', is there?  But it's there in 3.0.6-pre-2 as well, so maybe there's
something I'm just failing to understand.

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: Parameter substitution bug

[Bart Schaefer]

On Apr 29,  6:07pm, Peter Stephenson wrote:
> Subject: Parameter substitution bug
> Can anybody explain this?
> 
> % cat tst
> print $HOME/bin
> print ~/bin
> fdir='~/bin'
> print ${~fdir}
> setopt globsubst
> print $fdir
> % zsh -f ./tst
> /home/user2/pws/bin
> /home/user2/pws/bin
> ~/bin
> ~/bin
> 
> Why don't the last two give the same as the first two?

Because globsubst and ${~...} only do filename *generation* not filename
*expansion*.  It's always been like this ... you never noticed?

Tilde is not a glob character.

I always thought that using tilde as the turn-on-globsubst character was
a bad idea because it tends to make one think that tilde-expansion will
happen ... but there's really no other character available.

PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug

[Peter Stephenson]

"Bart Schaefer" wrote:
> On Apr 29,  6:07pm, Peter Stephenson wrote:
> > Subject: Parameter substitution bug
> > Can anybody explain this?
> > 
> > % cat tst
> > fdir='~/bin'
> > print ${~fdir}
> > % zsh -f ./tst
> > ~/bin
> 
> Because globsubst and ${~...} only do filename *generation* not filename
> *expansion*.  It's always been like this ... you never noticed?

No, that's not the answer, and it certainly hasn't always been like this.

1. Far from never noticing, I originally wrote the option (it had a
different name then) *specifically* to expand ~'s from variables, as csh
did.  The globbing stuff only got added in later on, then the name got
changed --- there was a note in the FAQ, but it became out of date some
time ago.  That's why the flag ~ was picked originally; it didn't get
changed when the option name did.  In fact, this is what it says in the
manual:

  Treat any characters resulting from parameter expansion as being
  eligible for file expansion and filename generation, and any
  characters resulting from command substitution as being eligible for
  filename generation.


2. I've discovered what the problem was, and it certainly looks
unintentional to me:

% fdir='~/bin'
% unsetopt extendedglob
% print ${~fdir}
~/bin
% setopt extendedglob
% print ${~fdir}
/home/user2/pws/bin

I take it this is universally regarded as a bug?  Parameter expansion flags
certainly shouldn't do that.

This has come about because tokenize() only tokenizes a ~ if extendedglob
is set.  This shouldn't be necessary --- the globbing code was specially
rewritten so that it only uses a tokenized ~, ^, etc. when extendedglob is
set.  That's because the shell can parse a function when extendedglob is
not set, only to have the option set when the function is executed for use
within the function, with the result that it doesn't work because the
parsed strings aren't properly tokenized.  I think we should try the
following patch --- and if it doesn't work, the problem is somewhere else
anyway; the lexer has happily been unconditionally tokenizing ~, # and ^
for some time now.

Note that I've made it so that ^, # and ~ are now tokenized when SHGLOB is
set --- the same argument applies here, except that this has the effect
that extendedglob is independent of shglob for retokenized characters;
since it always was for characters lexed properly, I don't see a problem
with that.  However, probably SHGLOB should be handled somewhere other than
in the lexer for the same reason that if it's set when a function is
parsed, parentheses won't work in the corresponding function even if it's
unset (e.g. by `emulate -L zsh').

This should apply to 3.0.6 as well.

--- Src/glob.c.eg	Wed Apr 21 16:18:09 1999
+++ Src/glob.c	Fri Apr 30 09:48:42 1999
@@ -3379,16 +3379,14 @@
 	    *t = Inang;
 	    *s = Outang;
 	    break;
-	case '^':
-	case '#':
-	case '~':
-	    if (unset(EXTENDEDGLOB))
-		break;
 	case '(':
 	case '|':
 	case ')':
 	    if (isset(SHGLOB))
 		break;
+	case '^':
+	case '#':
+	case '~':
 	case '[':
 	case ']':
 	case '*':


-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug

[Peter Stephenson]

"Bart Schaefer" wrote:
> On Apr 30, 10:14am, Peter Stephenson wrote:
> } Subject: PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug
> }
> } This should apply to 3.0.6 as well.
> 
> It didn't, because 3.0.x doesn't handle '[' and ']' the same way.  Is
> the following a correct patch, or is more extensive surgery required?

Sorry, missed the differences.  That looks fine.

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug

[Bart Schaefer]

On Apr 30, 10:14am, Peter Stephenson wrote:
} Subject: PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug
}
} "Bart Schaefer" wrote:
} > Because globsubst and ${~...} only do filename *generation* not filename
} > *expansion*.  It's always been like this ... you never noticed?
} 
} No, that's not the answer, and it certainly hasn't always been like this.

What am I thinking of, then?  Hmm ... back on 20aug98, I wrote in
zsh-workers/4342:

> I've just diffed the 3.0.5 multicomp against a much older one, and found
> that the original function used
> 
> 	eval "reply=($reply)"
> 
> where the current version uses
> 
> 	reply=(${~reply})
> 
> The former expands tildes, the latter does not.

and I went on to prove that black is white, and now I've just been killed
at the zebra crossing.

So when did this get broken, and how come nobody noticed seven months ago?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug

[Bart Schaefer]

On Apr 30, 10:14am, Peter Stephenson wrote:
} Subject: PATCH: 3.1.5++/3.0.6 Re: Parameter substitution bug
}
} This should apply to 3.0.6 as well.

It didn't, because 3.0.x doesn't handle '[' and ']' the same way.  Is
the following a correct patch, or is more extensive surgery required?

Index: Src/glob.c
===================================================================
--- glob.c	1999/04/28 05:21:34	1.1.1.5.2.3
+++ glob.c	1999/04/30 09:56:42
@@ -2203,16 +2203,14 @@
 	    *t = Inang;
 	    *s = Outang;
 	    break;
-	case '^':
-	case '#':
-	case '~':
-	    if (unset(EXTENDEDGLOB))
-		break;
 	case '(':
 	case '|':
 	case ')':
 	    if (isset(SHGLOB))
 		break;
+	case '^':
+	case '#':
+	case '~':
 	case '*':
 	case '?':
 	    for (t = ztokens; *t; t++)

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com