Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

[ There is a little patchlet below. I don't suggest using it to
  everyone just now, it's just that I can think about this better when 
  I have cod to play with. Didn't have the time for this yesterday. ]

Bart Schaefer wrote:

> Think about it this way:
> 
> 1. The expression "${(@)foo}" produces an array of quoted words.
> 
> 2. In the expression "${${(@)foo}}", the outer ${ ... } can include both
> flags in leading () and subscripts in trailing [].  The inner ${(@)foo}
> should *still* produce an array of quoted words, to which the flags and
> subscripts of the outer ${ ... } are applied.
> 
> 3. To work like the old zsh, the subscripts should apply *before* any
> split or join operation implied by the flags (or by the lack of a flag).
> The exception being the (P) flag, because it is defined to change the
> interpretation of the parameter's name and thus must "bind tighter" than
> anything including subscripts.
> 
> In pws-14+, at step (2), the inner expression does NOT produce an array
> unless the outer expression uses (@).  This is what I think is wrong.

Agreed (again) -- I mentioned this in 5841.

> I don't think it's possible to "pass down" from paramsubst-->multsub
> the knowledge of whether an array should be returned [except when (P)];
> it has to be "passed up," multsub<--prefork<--stringsubst<--paramsubst.

I don't understand the difference between making the the arrows point
to the left or the right here.

> Beyond those three steps, things get fuzzy.  In trying to reason about
> this without actually seeing it in action, I *believe* that it's OK if:
> 
> 4. *After* the subscripts are applied, the outer ${ ... } joins the
> quoted words into a single string, *unless*: the (@) flag is present
> or the [@] subscript was applied, and the (j) flag is not present.

This one is very important. If we make it this way (and the patch
below does that), it means that we still need the plethora of `(@)'
flags: with `foo' being an array "${${(@)foo}[1,2]}" selects the first 
two elements of it, joins them, and returns *one* string.
So, this doesn't help much to simplify the syntax needed in some of
the more common cases.

> I know that's still a change from the old behavior, so I can't be sure
> that it'll work out, but I *think* it will.
> 
> 5. Finally, the string is split again according to SHWORDSPLIT or the
> (s) flag.
> 
> 6. If there is a split, or if (@) or [@] was present, an array of quoted
> words is returned.  If there's another outer ${ ... }, go to step (3).

These two are done in a piece of code I didn't change anyway and were
only affected by the outcome of the call to multsub().
> 
> } With respect to the outer paramsubst()s this would make things
> } independent of whether the whole thing is in quotes or not, only the
> } inner paramsubst()s `control' if the outer ones work on an array
> } by testing if the thing is in quotes and if the `(@)' flag (or
> } something similar like the `s' flag or `=') is used.
> 
> As I just said, I don't *think* it should be independent of whether the
> whole thing is in quotes.  The quotes should affect what happens at (4),
> but they should *not* affect what happens at 2-->3 or 6-->3.

Obviously, I wasn't clear enough again. In terms of code: I wanted to
say that at the call to multsub(), the information about whether we
are in quotes should not be used. But of course, it should be used
after that -- as usual. Only make sure that we get correct information 
about the array-ness whether we are in quotes or not. And of course,
the quotes also affect the way the inner expression is expanded and
hence if it yields an array or not.

> Does that make sense?

Yes, and the behavior with the patch looks better than before to
me. We only need to discuss the `is there a way to avoid the need for
the many (@) flags' thing.

Bye
 Sven

--- os/subst.c	Wed Apr 14 11:58:36 1999
+++ Src/subst.c	Fri Apr 16 09:09:32 1999
@@ -245,35 +245,43 @@
  * the result is stored in *a. If `a' is zero a multiple word result is *
  * joined using sep or the IFS parameter if sep is zero and the result  *
  * is returned in *s.  The return value is true iff the expansion       *
- * resulted in an empty list                                            */
+ * resulted in an empty list.                                           *
+ * The mult_isarr variable is used by paramsubst() to tell if it yields *
+ * an array.                                                            */
+
+static int mult_isarr;
 
 /**/
 static int
 multsub(char **s, char ***a, int *isarr, char *sep)
 {
     LinkList foo;
-    int l;
+    int l, omi = mult_isarr;
     char **r, **p;
 
+    mult_isarr = 0;
     foo = newlinklist();
     addlinknode(foo, *s);
     prefork(foo, 0);
     if (errflag) {
 	if (isarr)
 	    *isarr = 0;
+	mult_isarr = omi;
 	return 0;
     }
-    if ((l = countlinknodes(foo)) > 1 || a) {
+    if ((l = countlinknodes(foo))) {
 	p = r = ncalloc((l + 1) * sizeof(char*));
 	while (nonempty(foo))
 	    *p++ = (char *)ugetnode(foo);
 	*p = NULL;
-	if (a) {
+	if (a && mult_isarr) {
 	    *a = r;
 	    *isarr = 1;
+	    mult_isarr = omi;
 	    return 0;
 	}
 	*s = sepjoin(r, NULL);
+	mult_isarr = omi;
 	return 0;
     }
     if (l)
@@ -282,6 +290,7 @@
 	*s = dupstring("");
     if (isarr)
 	*isarr = 0;
+    mult_isarr = omi;
     return !l;
 }
 
@@ -977,16 +986,12 @@
 	skipparens(*s, *s == Inpar ? Outpar : Outbrace, &s);
 	sav = *s;
 	*s = 0;
-	if (multsub(&val, ((!aspar && (!quoted || nojoin)) ? &aval : NULL),
-		    &isarr, NULL) &&
-	    quoted) {
+	if (multsub(&val, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
 	    isarr = -1;
 	    aval = alloc(sizeof(char *));
 	    aspar = 0;
 	} else if (aspar)
 	    idbeg = val;
-	if (isarr)
-	    isarr = -1;
 	copied = 1;
 	*s = sav;
 	v = (Value) NULL;
@@ -1465,6 +1470,7 @@
 	val = dupstring(buf);
 	isarr = 0;
     }
+    mult_isarr = isarr;
     if (isarr > 0 && !plan9 && (!aval || !aval[0])) {
 	val = dupstring("");
 	isarr = 0;

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

RE: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Andrej Borsenkow]

Unfortunately, I haven't followed this discussion closely - buy there is one
thing that actually nerves me.

It is *impossible* to split array elements. It is no joke. Arrays are joined
together before splitting - so, the result I get is much much different. As
example,

foo=(a:b x:y)
bor@itsrm2:~%> print -l ${(s/:/)foo}
a
b x
y

Where I would actually expect
a
b
x
y

Subsequent splitting on IFS does not help, as it can change "too much"

May be, nobody really needs it (I missed it time to time). And may be, there
is always workaround. But it is really really weird ...

/andrej

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 16, 11:48am, Andrej Borsenkow wrote:
} Subject: RE: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} It is *impossible* to split array elements. It is no joke. Arrays are joined
} together before splitting - so, the result I get is much much different.
} 
} Subsequent splitting on IFS does not help, as it can change "too much"
} 
} May be, nobody really needs it (I missed it time to time). And may be, there
} is always workaround. But it is really really weird ...

The obvious workaround is to explicitly join and then split:

zsh% foo=(a:b x:y)
zsh% print -l ${(j/:/s/:/)foo}
a
b
x
y

One might however argue that this looks odd:

zsh% print -l ${(s/:/j/:/)foo}
a
b
x
y

The flags are always applied in an internally-defined order, not in the
order they appear in the flag list.  But flag list ordering does matter
if you happen to repeat a flag:

zsh% print -l ${(s/:/s/ /)foo}
a:b
x:y

All that aside, a desirable (?) effect of the current interpretation is
that join reverses split (except for the initial array-to-scalar change):

zsh% print -l ${(j/:/)${(s/:/)foo}}
a:b x:y

With your interpretation, this would yield a:b:x:y.  Of course, since
split can't logically reverse join, maybe it's not necessary that join
be capable of reversing split.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Andrej Borsenkow]

>
> All that aside, a desirable (?) effect of the current interpretation is
> that join reverses split (except for the initial array-to-scalar change):
>

... as long as no SH_WORD_SPLIT is in effect. IFS splits may remove more
than IFS joins add.

> zsh% print -l ${(j/:/)${(s/:/)foo}}
> a:b x:y
>
> With your interpretation, this would yield a:b:x:y.

Don't laugh - it was exactly what I needed once. And I personally find
nothing wrong or illogical with it.

/andrej

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 16,  9:36am, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} Bart Schaefer wrote:
} 
} > I don't think it's possible to "pass down" from paramsubst-->multsub
} > the knowledge of whether an array should be returned [except when (P)];
} > it has to be "passed up," multsub<--prefork<--stringsubst<--paramsubst.
} 
} I don't understand the difference between making the the arrows point
} to the left or the right here.

It's a call stack, with caller on the left and the callee on the right.
Right-pointing arrows mean passed parameters, left-pointing arrows mean
returned value(s) [including "returned" through a static global].

} > 4. *After* the subscripts are applied, the outer ${ ... } joins the
} > quoted words into a single string, *unless*: the (@) flag is present
} > or the [@] subscript was applied, and the (j) flag is not present.
} 
} This one is very important. If we make it this way (and the patch
} below does that), it means that we still need the plethora of `(@)'

Yes, I know ...

} [...] the behavior with the patch looks better than before to
} me. We only need to discuss the `is there a way to avoid the need for
} the many (@) flags' thing.

I think the way to approach this may be to try to rewrite Peter's manual
section on parameter substitution rules to see how confusing it is when
joining does not occur.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Peter Stephenson]

Sven Wischnowsky wrote:
> 
> [ There is a little patchlet below. I don't suggest using it to
>   everyone just now, it's just that I can think about this better when 
>   I have cod to play with. Didn't have the time for this yesterday. ]

Is there any agreement on this patch (6046) to introduce the variable
mult_isarr to keep track of whether a nested parameter substitution returns
an array?  If so, what documentation changes are needed (which might help
tell everyone else what it's doing)?  And what about the other glitch that
was noticed, that a single word array is always turned into a scalar
(currently around line 1471 in subst.c), do we get rid of that for
consistency?  And do either of these require any changes to the existing
shell code in the distribution?

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

Peter Stephenson wrote:

> Sven Wischnowsky wrote:
> > 
> > [ There is a little patchlet below. I don't suggest using it to
> >   everyone just now, it's just that I can think about this better when 
> >   I have cod to play with. Didn't have the time for this yesterday. ]
> 
> Is there any agreement on this patch (6046) to introduce the variable
> mult_isarr to keep track of whether a nested parameter substitution returns
> an array?  If so, what documentation changes are needed (which might help
> tell everyone else what it's doing)?  And what about the other glitch that
> was noticed, that a single word array is always turned into a scalar
> (currently around line 1471 in subst.c), do we get rid of that for
> consistency?  And do either of these require any changes to the existing
> shell code in the distribution?

Ah, this was on my todo-list, too.

Quick description just to make sure that anyone knows what we are
talking about: without that patch "${${(@)a}[1,2]}" with `a' being 
an array, the subscript works on a *string*, the concatenated elements
of `a', even though there is a `(@)' flag. To get subscription on the
array elements, one has to add a second `(@)' flag in the outer
expansion. Another effect is that with the patch "${(@)${a}[1,2]}"
behaves like "${${a}[1,2]}". Without the patch this (somewhat
irritatingly) gives all elements of the array concatenated in one
string, because there the `(@)' made the inner expansion return an
array of one element.

Changes in the documentation that would be needed:

 Point 1.: the value returned is not always an array, only if the inner
    expansion would yield one (w.r.t to quoting and whatnot)
 The first `foo' example ("${(@)${foo}[1]}"): see above, the inner
    "${foo}" produces a string and subscripting yields `b'
 The second example ("${${(@)foo}[1]}"): produces `bar', because the
    inner "${(@)foo}" yields an array which is then subscripted

[Btw. point 2. is still wrong (multiple subscripts always work on the
result from previous subscripts, the parameter need not be an array).]

About the singleton-array-to-scalar-conversion: this irritated me,
too, the patch for 6046 sets mult_isarr earlier than I would have done 
it if this piece of code weren't there. I haven't played enough with
it to find out when exactly this is useful (if it is at all).

About changes needed in the shell code: I once had this patch in my
version, of course. I used as a normal working shell and the
completion functions at least didn't suffer from it. But they (almost?)
always use the `(@)' flag on every level and the behavior for that
wasn't changed. I haven't tried all example shell code stuff we have,
but a quick `grep @ ...' looks good (we don't use subscripts on nested 
parameter expansion very often, it seems).

Maybe Bart could give some more help here...

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On May 7,  1:51pm, Peter Stephenson wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} Sven Wischnowsky wrote:
} > 
} > [ There is a little patchlet below. I don't suggest using it to
} >   everyone just now, it's just that I can think about this better when 
} >   I have cod to play with. Didn't have the time for this yesterday. ]
} 
} Is there any agreement on this patch (6046) to introduce the variable
} mult_isarr to keep track of whether a nested parameter substitution returns
} an array?

I think it's more sensible behavior than what's happening now.  It's a
little less likely to break scripts written for 3.0.x, too.

} If so, what documentation changes are needed (which might help
} tell everyone else what it's doing)?

We'd have to revise the "parameter expansion rules" section that you
added.  In particular, the latter part of this:

  1. *Nested Substitution*
     If multiple nested `${...}' forms are present, substitution is
     performed from the inside outwards.  At each level, the
     substitution takes account of whether the current value is a
     scalar or an array, whether the whole substitution is in double
     quotes, and what flags are supplied to the current level of
     substitution; the flags are not propagated up to enclosing
     substitutions.  The value passed back to an enclosing substitution
     is always an array, which however will consist of one word if the
     value was not itself an array.  All the following steps take place
     where applicable at all levels of substitution.

Becomes (see Sven's note):
                                        ... the current level of
     substitution, just as if the nested substitution were the
     outermost.  The flags are not propagated up to enclosing
     substitutions; nor are arrays joined into strings, nor strings
     split into arrays, except as indicated by the flags at the
     current level as adjusted for quoting.  All the following steps
     take place where applicable at all levels of substitution.

I don't know whether we'd want to add something like:

     Note that, unless the (P) flag is present, the flags and any
     subscripts at each enclosing substitution apply directly to the
     value of the nested substitution.

Then as Sven mentioned:

  2. *Parameter Subscripting*
     If the value is a raw parameter reference with a subscript, such
     as ${VAR[3]}, the effect of subscripting is applied directly to
     the value of the parameter.  Subscripts are evaluated left to
     right; each second and succeeding subscript applies to the value
     yielded by the previous subscript.  Thus, if VAR is an array,
     ${VAR[1][2]} is the second character of the first word, but
     ${VAR[2,4][2]} is the entire third word (the second word of the
     range of words two through four of the original array).

Perhaps an example should note that

     ${foo[2,4][2]} == ${${foo[2,4]}[2]}		(No quotes!)
     ${${foo[2,4]}[2]} != "${${foo[2,4]}[2]}"		(Quotes matter)
     ${${foo[2,4]}[2]} == "${${(@)foo[2,4]}[2]}"	((@) matters)

And finally, the example changes Sven described.

} And what about the other glitch that was noticed, that a single word
} array is always turned into a scalar (currently around line 1471 in
} subst.c), do we get rid of that for consistency?

I tried deleting that and got a core dump elsewhere in subst.c.  There's
a loop over the array that assumes index 1 must not be null because it's
not possible (when the line-1471-code is present) to get there with a 1-
element array.  I haven't worked out whether there are other consequences
to stopping that loop sooner.

I think we should correct it, but there's almost certainly some reason
for it, perhaps now lost in the mists of time.

On May 7,  3:36pm, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} About changes needed in the shell code: I once had this patch in my
} version, of course. I used as a normal working shell and the
} completion functions at least didn't suffer from it. But they (almost?)
} always use the `(@)' flag on every level and the behavior for that
} wasn't changed. I haven't tried all example shell code stuff we have,
} but a quick `grep @ ...' looks good (we don't use subscripts on nested 
} parameter expansion very often, it seems).

Right.

The only good alternatives at this point are probably [a] go back to the
3.0 behavior of (@), or [b] leave it the way this mult_isarr patch leaves
it.  Can anyone think of something in between that makes sense?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

PATCH: mult_isarr documentation

[Peter Stephenson]

Here's what I have for the documentation for 6046 after Sven's and Bart's
suggestions.

I'm hoping to get the latest compinstall (and consequent compinit) changes
finished before producing pws-18.

--- Doc/Zsh/expn.yo.ma	Mon Apr 26 09:25:08 1999
+++ Doc/Zsh/expn.yo	Mon May 10 10:18:42 1999
@@ -717,18 +717,24 @@
 performed from the inside outwards.  At each level, the substitution takes
 account of whether the current value is a scalar or an array, whether the
 whole substitution is in double quotes, and what flags are supplied to the
-current level of substitution; the flags are not propagated up to enclosing
-substitutions.  The value passed back to an enclosing substitution is
-always an array, which however will consist of one word if the value was
-not itself an array.  All the following steps take place where applicable
-at all levels of substitution.
+current level of substitution, just as if the nested substitution were the
+outermost.  The flags are not propagated up to enclosing
+substitutions; the nested subsitution will return either a scalar or an
+array as determined by the flags, possibly adjusted for quoting.  All the
+following steps take place where applicable at all levels of substitution.
+Note that, unless the tt((P)) flag is present, the flags and any subscripts
+apply directly to the value of the nested substitution; for example, the
+expansion tt(${${foo}}) behaves exactly the same as tt(${foo}).
 )
 item(tt(2.) em(Parameter Subscripting))(
 If the value is a raw parameter reference with a subscript, such as
 tt(${)var(var)tt([3]}), the effect of subscripting is applied directly to
-the parameter.  If the parameter is an array, any second subscript,
-indexing on the character in the word, may appear,
-e.g. tt(${)var(var)tt([1][2]}).
+the parameter.  Subscripts are evaluated left to right; subsequent
+subscripts apply to the scalar or array value yielded by the previous
+subscript.  Thus if tt(var) is an array, tt(${var[1][2]}) is the second
+character of the first word, but tt(${var[2,4][2]}) is the entire third
+word (the second word of the range of words two through four of the
+original array).  Any number of subscripts may appear.
 )
 item(tt(3.) em(Parameter Name Replacement))(
 The effect of any tt((P)) flag, which treats the value so far as a
@@ -746,6 +752,10 @@
 Any remaining subscript (i.e. of a nested substitution) is evaluated at
 this point, based on whether the value is an array or a scalar; if it was
 an array, a second subscript for the character in the word may also appear.
+Note that tt(${foo[2,4][2]}) is thus equivalent to tt(${${foo[2,4]}[2]})
+and also to tt("${${(@)foo[2,4]}[2]}") (the nested substitution returns an
+array in both cases), but not to tt("${${foo[2,4]}[2]}") (the nested
+substitution returns a scalar because of the quotes).
 )
 item(tt(6.) em(Modifiers))(
 Any modifiers, as specified by a trailing tt(#), tt(%), tt(/)
@@ -798,20 +808,17 @@
 
 startitem()
 item(tt("${(@)${foo}[1]}"))(
-This produces the result tt(bar baz).  First, the inner substitution
+This produces the result tt(b).  First, the inner substitution
 tt("${foo}"), which has no array (tt(@)) flag, produces a single word
-result.  The outer substitution tt("${(@)...[1]}") acts on this result as
-if it were a one word array, because of the array flag, so the result is
-just that single word.
+result tt("bar baz").  The outer substitution tt("${(@)...[1]}") detects
+that this is a scalar, so that (despite the tt((@)) flag) the subscript
+picks the first character. 
 )
 item(tt("${${(@)foo}[1]}"))(
-The produces the result tt(b).  In this case, the inner substitution
+The produces the result tt(bar).  In this case, the inner substitution
 tt("${(@)foo}") produces the array tt(LPAR()bar baz)tt(RPAR()).  The outer
-substitution tt("${...[1]}"), however, has no array flag, so that it joins
-the array it has to a single word and indexes as if it were a string.  Note
-that this is not identical to the case tt("${foo[1]}"), since here the
-expression tt(foo[1]) is recognised immediately as an index into an array,
-so that the result in that case is tt(bar).
+substitution tt("${...[1]}") detects that this is an array and picks the
+first word.  This is similar to the simple case tt("${foo[1]}").
 )
 enditem()
 
-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy