Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

[ There is a little patchlet below. I don't suggest using it to
  everyone just now, it's just that I can think about this better when 
  I have cod to play with. Didn't have the time for this yesterday. ]

Bart Schaefer wrote:

> Think about it this way:
> 
> 1. The expression "${(@)foo}" produces an array of quoted words.
> 
> 2. In the expression "${${(@)foo}}", the outer ${ ... } can include both
> flags in leading () and subscripts in trailing [].  The inner ${(@)foo}
> should *still* produce an array of quoted words, to which the flags and
> subscripts of the outer ${ ... } are applied.
> 
> 3. To work like the old zsh, the subscripts should apply *before* any
> split or join operation implied by the flags (or by the lack of a flag).
> The exception being the (P) flag, because it is defined to change the
> interpretation of the parameter's name and thus must "bind tighter" than
> anything including subscripts.
> 
> In pws-14+, at step (2), the inner expression does NOT produce an array
> unless the outer expression uses (@).  This is what I think is wrong.

Agreed (again) -- I mentioned this in 5841.

> I don't think it's possible to "pass down" from paramsubst-->multsub
> the knowledge of whether an array should be returned [except when (P)];
> it has to be "passed up," multsub<--prefork<--stringsubst<--paramsubst.

I don't understand the difference between making the the arrows point
to the left or the right here.

> Beyond those three steps, things get fuzzy.  In trying to reason about
> this without actually seeing it in action, I *believe* that it's OK if:
> 
> 4. *After* the subscripts are applied, the outer ${ ... } joins the
> quoted words into a single string, *unless*: the (@) flag is present
> or the [@] subscript was applied, and the (j) flag is not present.

This one is very important. If we make it this way (and the patch
below does that), it means that we still need the plethora of `(@)'
flags: with `foo' being an array "${${(@)foo}[1,2]}" selects the first 
two elements of it, joins them, and returns *one* string.
So, this doesn't help much to simplify the syntax needed in some of
the more common cases.

> I know that's still a change from the old behavior, so I can't be sure
> that it'll work out, but I *think* it will.
> 
> 5. Finally, the string is split again according to SHWORDSPLIT or the
> (s) flag.
> 
> 6. If there is a split, or if (@) or [@] was present, an array of quoted
> words is returned.  If there's another outer ${ ... }, go to step (3).

These two are done in a piece of code I didn't change anyway and were
only affected by the outcome of the call to multsub().
> 
> } With respect to the outer paramsubst()s this would make things
> } independent of whether the whole thing is in quotes or not, only the
> } inner paramsubst()s `control' if the outer ones work on an array
> } by testing if the thing is in quotes and if the `(@)' flag (or
> } something similar like the `s' flag or `=') is used.
> 
> As I just said, I don't *think* it should be independent of whether the
> whole thing is in quotes.  The quotes should affect what happens at (4),
> but they should *not* affect what happens at 2-->3 or 6-->3.

Obviously, I wasn't clear enough again. In terms of code: I wanted to
say that at the call to multsub(), the information about whether we
are in quotes should not be used. But of course, it should be used
after that -- as usual. Only make sure that we get correct information 
about the array-ness whether we are in quotes or not. And of course,
the quotes also affect the way the inner expression is expanded and
hence if it yields an array or not.

> Does that make sense?

Yes, and the behavior with the patch looks better than before to
me. We only need to discuss the `is there a way to avoid the need for
the many (@) flags' thing.

Bye
 Sven

--- os/subst.c	Wed Apr 14 11:58:36 1999
+++ Src/subst.c	Fri Apr 16 09:09:32 1999
@@ -245,35 +245,43 @@
  * the result is stored in *a. If `a' is zero a multiple word result is *
  * joined using sep or the IFS parameter if sep is zero and the result  *
  * is returned in *s.  The return value is true iff the expansion       *
- * resulted in an empty list                                            */
+ * resulted in an empty list.                                           *
+ * The mult_isarr variable is used by paramsubst() to tell if it yields *
+ * an array.                                                            */
+
+static int mult_isarr;
 
 /**/
 static int
 multsub(char **s, char ***a, int *isarr, char *sep)
 {
     LinkList foo;
-    int l;
+    int l, omi = mult_isarr;
     char **r, **p;
 
+    mult_isarr = 0;
     foo = newlinklist();
     addlinknode(foo, *s);
     prefork(foo, 0);
     if (errflag) {
 	if (isarr)
 	    *isarr = 0;
+	mult_isarr = omi;
 	return 0;
     }
-    if ((l = countlinknodes(foo)) > 1 || a) {
+    if ((l = countlinknodes(foo))) {
 	p = r = ncalloc((l + 1) * sizeof(char*));
 	while (nonempty(foo))
 	    *p++ = (char *)ugetnode(foo);
 	*p = NULL;
-	if (a) {
+	if (a && mult_isarr) {
 	    *a = r;
 	    *isarr = 1;
+	    mult_isarr = omi;
 	    return 0;
 	}
 	*s = sepjoin(r, NULL);
+	mult_isarr = omi;
 	return 0;
     }
     if (l)
@@ -282,6 +290,7 @@
 	*s = dupstring("");
     if (isarr)
 	*isarr = 0;
+    mult_isarr = omi;
     return !l;
 }
 
@@ -977,16 +986,12 @@
 	skipparens(*s, *s == Inpar ? Outpar : Outbrace, &s);
 	sav = *s;
 	*s = 0;
-	if (multsub(&val, ((!aspar && (!quoted || nojoin)) ? &aval : NULL),
-		    &isarr, NULL) &&
-	    quoted) {
+	if (multsub(&val, (aspar ? NULL : &aval), &isarr, NULL) && quoted) {
 	    isarr = -1;
 	    aval = alloc(sizeof(char *));
 	    aspar = 0;
 	} else if (aspar)
 	    idbeg = val;
-	if (isarr)
-	    isarr = -1;
 	copied = 1;
 	*s = sav;
 	v = (Value) NULL;
@@ -1465,6 +1470,7 @@
 	val = dupstring(buf);
 	isarr = 0;
     }
+    mult_isarr = isarr;
     if (isarr > 0 && !plan9 && (!aval || !aval[0])) {
 	val = dupstring("");
 	isarr = 0;

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

RE: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Andrej Borsenkow]

Unfortunately, I haven't followed this discussion closely - buy there is one
thing that actually nerves me.

It is *impossible* to split array elements. It is no joke. Arrays are joined
together before splitting - so, the result I get is much much different. As
example,

foo=(a:b x:y)
bor@itsrm2:~%> print -l ${(s/:/)foo}
a
b x
y

Where I would actually expect
a
b
x
y

Subsequent splitting on IFS does not help, as it can change "too much"

May be, nobody really needs it (I missed it time to time). And may be, there
is always workaround. But it is really really weird ...

/andrej

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 16, 11:48am, Andrej Borsenkow wrote:
} Subject: RE: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} It is *impossible* to split array elements. It is no joke. Arrays are joined
} together before splitting - so, the result I get is much much different.
} 
} Subsequent splitting on IFS does not help, as it can change "too much"
} 
} May be, nobody really needs it (I missed it time to time). And may be, there
} is always workaround. But it is really really weird ...

The obvious workaround is to explicitly join and then split:

zsh% foo=(a:b x:y)
zsh% print -l ${(j/:/s/:/)foo}
a
b
x
y

One might however argue that this looks odd:

zsh% print -l ${(s/:/j/:/)foo}
a
b
x
y

The flags are always applied in an internally-defined order, not in the
order they appear in the flag list.  But flag list ordering does matter
if you happen to repeat a flag:

zsh% print -l ${(s/:/s/ /)foo}
a:b
x:y

All that aside, a desirable (?) effect of the current interpretation is
that join reverses split (except for the initial array-to-scalar change):

zsh% print -l ${(j/:/)${(s/:/)foo}}
a:b x:y

With your interpretation, this would yield a:b:x:y.  Of course, since
split can't logically reverse join, maybe it's not necessary that join
be capable of reversing split.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Andrej Borsenkow]

>
> All that aside, a desirable (?) effect of the current interpretation is
> that join reverses split (except for the initial array-to-scalar change):
>

... as long as no SH_WORD_SPLIT is in effect. IFS splits may remove more
than IFS joins add.

> zsh% print -l ${(j/:/)${(s/:/)foo}}
> a:b x:y
>
> With your interpretation, this would yield a:b:x:y.

Don't laugh - it was exactly what I needed once. And I personally find
nothing wrong or illogical with it.

/andrej

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 16,  9:36am, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} Bart Schaefer wrote:
} 
} > I don't think it's possible to "pass down" from paramsubst-->multsub
} > the knowledge of whether an array should be returned [except when (P)];
} > it has to be "passed up," multsub<--prefork<--stringsubst<--paramsubst.
} 
} I don't understand the difference between making the the arrows point
} to the left or the right here.

It's a call stack, with caller on the left and the callee on the right.
Right-pointing arrows mean passed parameters, left-pointing arrows mean
returned value(s) [including "returned" through a static global].

} > 4. *After* the subscripts are applied, the outer ${ ... } joins the
} > quoted words into a single string, *unless*: the (@) flag is present
} > or the [@] subscript was applied, and the (j) flag is not present.
} 
} This one is very important. If we make it this way (and the patch
} below does that), it means that we still need the plethora of `(@)'

Yes, I know ...

} [...] the behavior with the patch looks better than before to
} me. We only need to discuss the `is there a way to avoid the need for
} the many (@) flags' thing.

I think the way to approach this may be to try to rewrite Peter's manual
section on parameter substitution rules to see how confusing it is when
joining does not occur.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Peter Stephenson]

Sven Wischnowsky wrote:
> 
> [ There is a little patchlet below. I don't suggest using it to
>   everyone just now, it's just that I can think about this better when 
>   I have cod to play with. Didn't have the time for this yesterday. ]

Is there any agreement on this patch (6046) to introduce the variable
mult_isarr to keep track of whether a nested parameter substitution returns
an array?  If so, what documentation changes are needed (which might help
tell everyone else what it's doing)?  And what about the other glitch that
was noticed, that a single word array is always turned into a scalar
(currently around line 1471 in subst.c), do we get rid of that for
consistency?  And do either of these require any changes to the existing
shell code in the distribution?

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

Peter Stephenson wrote:

> Sven Wischnowsky wrote:
> > 
> > [ There is a little patchlet below. I don't suggest using it to
> >   everyone just now, it's just that I can think about this better when 
> >   I have cod to play with. Didn't have the time for this yesterday. ]
> 
> Is there any agreement on this patch (6046) to introduce the variable
> mult_isarr to keep track of whether a nested parameter substitution returns
> an array?  If so, what documentation changes are needed (which might help
> tell everyone else what it's doing)?  And what about the other glitch that
> was noticed, that a single word array is always turned into a scalar
> (currently around line 1471 in subst.c), do we get rid of that for
> consistency?  And do either of these require any changes to the existing
> shell code in the distribution?

Ah, this was on my todo-list, too.

Quick description just to make sure that anyone knows what we are
talking about: without that patch "${${(@)a}[1,2]}" with `a' being 
an array, the subscript works on a *string*, the concatenated elements
of `a', even though there is a `(@)' flag. To get subscription on the
array elements, one has to add a second `(@)' flag in the outer
expansion. Another effect is that with the patch "${(@)${a}[1,2]}"
behaves like "${${a}[1,2]}". Without the patch this (somewhat
irritatingly) gives all elements of the array concatenated in one
string, because there the `(@)' made the inner expansion return an
array of one element.

Changes in the documentation that would be needed:

 Point 1.: the value returned is not always an array, only if the inner
    expansion would yield one (w.r.t to quoting and whatnot)
 The first `foo' example ("${(@)${foo}[1]}"): see above, the inner
    "${foo}" produces a string and subscripting yields `b'
 The second example ("${${(@)foo}[1]}"): produces `bar', because the
    inner "${(@)foo}" yields an array which is then subscripted

[Btw. point 2. is still wrong (multiple subscripts always work on the
result from previous subscripts, the parameter need not be an array).]

About the singleton-array-to-scalar-conversion: this irritated me,
too, the patch for 6046 sets mult_isarr earlier than I would have done 
it if this piece of code weren't there. I haven't played enough with
it to find out when exactly this is useful (if it is at all).

About changes needed in the shell code: I once had this patch in my
version, of course. I used as a normal working shell and the
completion functions at least didn't suffer from it. But they (almost?)
always use the `(@)' flag on every level and the behavior for that
wasn't changed. I haven't tried all example shell code stuff we have,
but a quick `grep @ ...' looks good (we don't use subscripts on nested 
parameter expansion very often, it seems).

Maybe Bart could give some more help here...

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On May 7,  1:51pm, Peter Stephenson wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} Sven Wischnowsky wrote:
} > 
} > [ There is a little patchlet below. I don't suggest using it to
} >   everyone just now, it's just that I can think about this better when 
} >   I have cod to play with. Didn't have the time for this yesterday. ]
} 
} Is there any agreement on this patch (6046) to introduce the variable
} mult_isarr to keep track of whether a nested parameter substitution returns
} an array?

I think it's more sensible behavior than what's happening now.  It's a
little less likely to break scripts written for 3.0.x, too.

} If so, what documentation changes are needed (which might help
} tell everyone else what it's doing)?

We'd have to revise the "parameter expansion rules" section that you
added.  In particular, the latter part of this:

  1. *Nested Substitution*
     If multiple nested `${...}' forms are present, substitution is
     performed from the inside outwards.  At each level, the
     substitution takes account of whether the current value is a
     scalar or an array, whether the whole substitution is in double
     quotes, and what flags are supplied to the current level of
     substitution; the flags are not propagated up to enclosing
     substitutions.  The value passed back to an enclosing substitution
     is always an array, which however will consist of one word if the
     value was not itself an array.  All the following steps take place
     where applicable at all levels of substitution.

Becomes (see Sven's note):
                                        ... the current level of
     substitution, just as if the nested substitution were the
     outermost.  The flags are not propagated up to enclosing
     substitutions; nor are arrays joined into strings, nor strings
     split into arrays, except as indicated by the flags at the
     current level as adjusted for quoting.  All the following steps
     take place where applicable at all levels of substitution.

I don't know whether we'd want to add something like:

     Note that, unless the (P) flag is present, the flags and any
     subscripts at each enclosing substitution apply directly to the
     value of the nested substitution.

Then as Sven mentioned:

  2. *Parameter Subscripting*
     If the value is a raw parameter reference with a subscript, such
     as ${VAR[3]}, the effect of subscripting is applied directly to
     the value of the parameter.  Subscripts are evaluated left to
     right; each second and succeeding subscript applies to the value
     yielded by the previous subscript.  Thus, if VAR is an array,
     ${VAR[1][2]} is the second character of the first word, but
     ${VAR[2,4][2]} is the entire third word (the second word of the
     range of words two through four of the original array).

Perhaps an example should note that

     ${foo[2,4][2]} == ${${foo[2,4]}[2]}		(No quotes!)
     ${${foo[2,4]}[2]} != "${${foo[2,4]}[2]}"		(Quotes matter)
     ${${foo[2,4]}[2]} == "${${(@)foo[2,4]}[2]}"	((@) matters)

And finally, the example changes Sven described.

} And what about the other glitch that was noticed, that a single word
} array is always turned into a scalar (currently around line 1471 in
} subst.c), do we get rid of that for consistency?

I tried deleting that and got a core dump elsewhere in subst.c.  There's
a loop over the array that assumes index 1 must not be null because it's
not possible (when the line-1471-code is present) to get there with a 1-
element array.  I haven't worked out whether there are other consequences
to stopping that loop sooner.

I think we should correct it, but there's almost certainly some reason
for it, perhaps now lost in the mists of time.

On May 7,  3:36pm, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} About changes needed in the shell code: I once had this patch in my
} version, of course. I used as a normal working shell and the
} completion functions at least didn't suffer from it. But they (almost?)
} always use the `(@)' flag on every level and the behavior for that
} wasn't changed. I haven't tried all example shell code stuff we have,
} but a quick `grep @ ...' looks good (we don't use subscripts on nested 
} parameter expansion very often, it seems).

Right.

The only good alternatives at this point are probably [a] go back to the
3.0 behavior of (@), or [b] leave it the way this mult_isarr patch leaves
it.  Can anyone think of something in between that makes sense?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

PATCH: mult_isarr documentation

[Peter Stephenson]

Here's what I have for the documentation for 6046 after Sven's and Bart's
suggestions.

I'm hoping to get the latest compinstall (and consequent compinit) changes
finished before producing pws-18.

--- Doc/Zsh/expn.yo.ma	Mon Apr 26 09:25:08 1999
+++ Doc/Zsh/expn.yo	Mon May 10 10:18:42 1999
@@ -717,18 +717,24 @@
 performed from the inside outwards.  At each level, the substitution takes
 account of whether the current value is a scalar or an array, whether the
 whole substitution is in double quotes, and what flags are supplied to the
-current level of substitution; the flags are not propagated up to enclosing
-substitutions.  The value passed back to an enclosing substitution is
-always an array, which however will consist of one word if the value was
-not itself an array.  All the following steps take place where applicable
-at all levels of substitution.
+current level of substitution, just as if the nested substitution were the
+outermost.  The flags are not propagated up to enclosing
+substitutions; the nested subsitution will return either a scalar or an
+array as determined by the flags, possibly adjusted for quoting.  All the
+following steps take place where applicable at all levels of substitution.
+Note that, unless the tt((P)) flag is present, the flags and any subscripts
+apply directly to the value of the nested substitution; for example, the
+expansion tt(${${foo}}) behaves exactly the same as tt(${foo}).
 )
 item(tt(2.) em(Parameter Subscripting))(
 If the value is a raw parameter reference with a subscript, such as
 tt(${)var(var)tt([3]}), the effect of subscripting is applied directly to
-the parameter.  If the parameter is an array, any second subscript,
-indexing on the character in the word, may appear,
-e.g. tt(${)var(var)tt([1][2]}).
+the parameter.  Subscripts are evaluated left to right; subsequent
+subscripts apply to the scalar or array value yielded by the previous
+subscript.  Thus if tt(var) is an array, tt(${var[1][2]}) is the second
+character of the first word, but tt(${var[2,4][2]}) is the entire third
+word (the second word of the range of words two through four of the
+original array).  Any number of subscripts may appear.
 )
 item(tt(3.) em(Parameter Name Replacement))(
 The effect of any tt((P)) flag, which treats the value so far as a
@@ -746,6 +752,10 @@
 Any remaining subscript (i.e. of a nested substitution) is evaluated at
 this point, based on whether the value is an array or a scalar; if it was
 an array, a second subscript for the character in the word may also appear.
+Note that tt(${foo[2,4][2]}) is thus equivalent to tt(${${foo[2,4]}[2]})
+and also to tt("${${(@)foo[2,4]}[2]}") (the nested substitution returns an
+array in both cases), but not to tt("${${foo[2,4]}[2]}") (the nested
+substitution returns a scalar because of the quotes).
 )
 item(tt(6.) em(Modifiers))(
 Any modifiers, as specified by a trailing tt(#), tt(%), tt(/)
@@ -798,20 +808,17 @@
 
 startitem()
 item(tt("${(@)${foo}[1]}"))(
-This produces the result tt(bar baz).  First, the inner substitution
+This produces the result tt(b).  First, the inner substitution
 tt("${foo}"), which has no array (tt(@)) flag, produces a single word
-result.  The outer substitution tt("${(@)...[1]}") acts on this result as
-if it were a one word array, because of the array flag, so the result is
-just that single word.
+result tt("bar baz").  The outer substitution tt("${(@)...[1]}") detects
+that this is a scalar, so that (despite the tt((@)) flag) the subscript
+picks the first character. 
 )
 item(tt("${${(@)foo}[1]}"))(
-The produces the result tt(b).  In this case, the inner substitution
+The produces the result tt(bar).  In this case, the inner substitution
 tt("${(@)foo}") produces the array tt(LPAR()bar baz)tt(RPAR()).  The outer
-substitution tt("${...[1]}"), however, has no array flag, so that it joins
-the array it has to a single word and indexes as if it were a string.  Note
-that this is not identical to the case tt("${foo[1]}"), since here the
-expression tt(foo[1]) is recognised immediately as an index into an array,
-so that the result in that case is tt(bar).
+substitution tt("${...[1]}") detects that this is an array and picks the
+first word.  This is similar to the simple case tt("${foo[1]}").
 )
 enditem()
 
-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

Bart Schaefer wrote:

> The only good alternatives at this point are probably [a] go back to the
> 3.0 behavior of (@), or [b] leave it the way this mult_isarr patch leaves
> it.  Can anyone think of something in between that makes sense?

It just occured to me that I didn't say that I would be in favor of
using the patch. I.e. to keep the things we have done to the `(@)'
stuff plus the patch. I consider the previous patches important,
because it allows one to distinguish arrays with one element from
strings and this last patch made things more consistent.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 15,  2:08pm, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} We have this call chain:
} 
}    A paramsubst(), calls
}    B    multsub(), calls
}    C       paramsubst()
} 
} Now, we could make the multsub() save (and at the end restore) the
} value of, say `static int mult_isarr'. It then sets it to zero and
} calls paramsubst() (via prefork(), as usual). At the end of
} paramsubst() we set `mult_isarr = isarr'.

I actually tried this last night -- although I had paramsubst() set the
static to 0 on the way in; multsub() just saved and restored it -- and
it worked for some cases but not others, so I decided I was missing a
subtlety somewhere and threw it out.

} After the prefork(), multsub() now knows that the list it gets was an
} array expression if `mult_isarr != 0'.

Right; this is interesting only when there's exactly one linknode in the
list returned by prefork().  (Hmm, or is it also interesting for zero
linknodes?  Should an empty array behave the same as an unset parameter,
or not?  Ah, no, it should not -- "$@" goes away entirely when $# == 0,
but "$*" yields the empty string.)

} This was the first part of what I meant -- getting information about
} the array'ness. With the other part I meant the call to multsub() we
} are talking about here (line 980). There we have to decide if we give
} `&aval' or `NULL' to multsub() where it is used to decide if the
} joining should be done -- the problem to solve, as you said.

Actually, that's NOT the problem to solve, or at least I didn't think so.
I think the code was correct back when `aspar' was the sole determinant
of whether NULL was passed in.  It's `mult_isarr', as you call it, that
should otherwise determine whether multsub() does a join.

} > No, that's too confusing and not necessary.  The only real issue is the
} > "precedence" if you will, of subscripting vs. joining.

The foregoing is the important point; I was confused here:

} > We "solved" the problem of subscripting scalars that were
} > accidentally treated as arrays, by forcing all quoted arrays to
} > be taken as scalars; I think we need to back off from that and
} > concentrate on not accidentally treating scalars as arrays.
} 
} I don't think I understand this. And I thought I kept it from arrays
} being accidentally being treated as scalars.

Um, yes; that's probably the subtlety that I missed.

} From your first mail about this I got the impression that you wanted
} nested expressions to keep their knowledge about whether they are
} arrays or not, so that (with foo being an array) in "${${${${(@)foo}}}[1]}"
} the outer paramsubst() (A) gets notified by multsub() (B) that the
} inner paramsubst() (C) returned an array and the subscripting gives
} the first array element.

Yes, that's the general idea.  I'm not sure how many levels of ${ } the
array-ness should propagate through, except that I currently believe it
should be at least one more level than it currently does.

Think about it this way:

1. The expression "${(@)foo}" produces an array of quoted words.

2. In the expression "${${(@)foo}}", the outer ${ ... } can include both
flags in leading () and subscripts in trailing [].  The inner ${(@)foo}
should *still* produce an array of quoted words, to which the flags and
subscripts of the outer ${ ... } are applied.

3. To work like the old zsh, the subscripts should apply *before* any
split or join operation implied by the flags (or by the lack of a flag).
The exception being the (P) flag, because it is defined to change the
interpretation of the parameter's name and thus must "bind tighter" than
anything including subscripts.

In pws-14+, at step (2), the inner expression does NOT produce an array
unless the outer expression uses (@).  This is what I think is wrong.
I don't think it's possible to "pass down" from paramsubst-->multsub
the knowledge of whether an array should be returned [except when (P)];
it has to be "passed up," multsub<--prefork<--stringsubst<--paramsubst.

Beyond those three steps, things get fuzzy.  In trying to reason about
this without actually seeing it in action, I *believe* that it's OK if:

4. *After* the subscripts are applied, the outer ${ ... } joins the
quoted words into a single string, *unless*: the (@) flag is present
or the [@] subscript was applied, and the (j) flag is not present.

I know that's still a change from the old behavior, so I can't be sure
that it'll work out, but I *think* it will.

5. Finally, the string is split again according to SHWORDSPLIT or the
(s) flag.

6. If there is a split, or if (@) or [@] was present, an array of quoted
words is returned.  If there's another outer ${ ... }, go to step (3).

} With respect to the outer paramsubst()s this would make things
} independent of whether the whole thing is in quotes or not, only the
} inner paramsubst()s `control' if the outer ones work on an array
} by testing if the thing is in quotes and if the `(@)' flag (or
} something similar like the `s' flag or `=') is used.

As I just said, I don't *think* it should be independent of whether the
whole thing is in quotes.  The quotes should affect what happens at (4),
but they should *not* affect what happens at 2-->3 or 6-->3.

Does that make sense?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

Bart Schaefer wrote:

> In what direction is the data flowing at this point?  Up or down the call
> stack?  I think you mean up; but paramsubst() is of course the caller of
> multsub() as well ...

I was slightly confused by this up/down thing, too. Again:

We have this call chain:

   A paramsubst(), calls
   B    multsub(), calls
   C       paramsubst()

Now, we could make the multsub() save (and at the end restore) the
value of, say `static int mult_isarr'. It then sets it to zero and
calls paramsubst() (via prefork(), as usual). At the end of
paramsubst() we set `mult_isarr = isarr'.

After the prefork(), multsub() now knows that the list it gets was an
array expression if `mult_isarr != 0'.

This was the first part of what I meant -- getting information about
the array'ness. With the other part I meant the call to multsub() we
are talking about here (line 980). There we have to decide if we give
`&aval' or `NULL' to multsub() where it is used to decide if the
joining should be done -- the problem to solve, as you said.

> Ideally, I think, it should still happen after, but the (non-)array-ness
> of whatever comes back from prefork() should be propagated up through
> isarr by multsub() -- which brings us back to the static global.

And this is what I meant (I just think less in terms of `isarr' then
in terms of `aval' -- and deciding when to give it to multsub()).

> No, that's too confusing and not necessary.  The only real issue is the
> "precedence" if you will, of subscripting vs. joining.  We "solved" the
> problem of subscripting scalars that were accidentally treated as arrays,
> by forcing all quoted arrays to be taken as scalars; I think we need to
> back off from that and concentrate on not accidentally treating scalars
> as arrays.

I don't think I understand this. And I thought I kept it from arrays
being accidentally being treated as scalars.

>From your first mail about this I got the impression that you wanted
nested expressions to keep their knowledge about whether they are
arrays or not, so that (with foo being an array) in "${${${${(@)foo}}}[1]}"
the outer paramsubst() (A) gets notified by multsub() (B) that the
inner paramsubst() (C) returned an array and the subscripting gives
the first array element. This `notification' would be done by having
multsub() use `&aval' instead of `&val' and by setting `*isarr', of
course.
With respect to the outer paramsubst()s this would make things
independent of whether the whole thing is in quotes or not, only the
inner paramsubst()s `control' if the outer ones work on an array
by testing if the thing is in quotes and if the `(@)' flag (or
something similar like the `s' flag or `=') is used.

Hope this makes it clearer.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 15,  8:49am, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} Bart Schaefer wrote:
} 
} > effect when the array has more than one element:
} > 
} > 	foo=(xa yb zc)
} > 	print -l "${${(@)foo}[1]}" "${(@)${foo}[1]}"
} > 
} > Zsh 3.0.5 prints
} > 
} > 	xa
} > 	x
} > 
} > But 3.1.5-pws-14 prints
} > 
} > 	x
} > 	xa
} 
} (Do you really get `xa' in the last case?)

I really did, but with a just-compiled -pws-15, I get

    x
    xa yb zc

so I'm not sure what was up.  In any case, this is just as bad.

} This (and the need to repeat the `(@)') comes from the fact that
} multsub() doesn't get any information about where the words came
} from. I.e. if they are an `array' (even if only one string) or not.
} I already said this at least once.

Yes, I know, but ...

} To solve this better we would need a way to make `paramsubst()' notify
} `multsub()' if the thing is an array.

In what direction is the data flowing at this point?  Up or down the call
stack?  I think you mean up; but paramsubst() is of course the caller of
multsub() as well ...

} This could be done by using a subst.c-global variable that is set in
} multsub() and paramsubst() and tested in multsub() after prefork(), of
} course.

Yes.  This is a little sloppy, but aside from completely rewriting the
whole multsub->prefork->stringsubst->paramsubst call chain, I think it
is the only way.  Unless we can stick something in the linklist that is
passed back, but that's probably even sloppier.

} And then we would have to decide when we want to accept an array at
} the call of multsub().

I've lost the direction of data flow again.  Do you mean accept an array
when prefork() returns to multsub()?  Or are you somehow talking about
what paramsubst() passes down to multsub()?

} Always, if the sub-expression is an array? Even if the whole thing
} is in double quotes? If it is in double quotes and we don't want it
} always, when?

I think what you're getting at is, when the expansion is in double
quotes, how soon do we perform sepjoin() on the words being passed up
from a nested substitution?

Right now, the answer is "always join in multsub(), unless the (@) flag
appears at the same level where the join would occur."  As best I can
tell -- and I have to go all the way back to 3.0.5, because the (P) flag
stuff has muddled it -- the answer used to be "never join in multsub()
unless performing a non-array assignment."  In both cases, still "do the
join in paramsubst() iff multsub() [via &isarr] said that it returned an
array, and the (@) flag does not appear."

The subtle difference comes in at this point, around subst.c:1057 in
pws-15:

              pm = createparam(nulstring, isarr ? PM_ARRAY : PM_SCALAR);
              if (isarr)
                  pm->u.arr = aval;
              else
                  pm->u.str = val;

When the join happens in multsub(), isarr is zero at that line, and we
create a dummy scalar parameter instead of a dummy array.  The code goes
on:

            v = (Value) hcalloc(sizeof *v);
            v->isarr = isarr;
            v->pm = pm;
            v->b = -1;
            if (getindex(&s, v) || s == os)
                break;

Because we have a dummy scalar, the interpretation of getindex() is
completely changed, and we see the effect I reported above.  The crucial
bit is whether the join happens before or after that getindex() call.
In all past versions of zsh, it happened *after*.

Ideally, I think, it should still happen after, but the (non-)array-ness
of whatever comes back from prefork() should be propagated up through
isarr by multsub() -- which brings us back to the static global.

} Hm, maybe when the `(@)' flag is given or the things came from an
} array. But then there would be no way to make an array be treated as
} a scalar further down up then by using the `(j:...:)' flag. But then
} again, this may be ok.

I think I know what you mean, but "farther down up" isn't helping.

} Or maybe we make the inner expression be taken as an array if it is a
} parameter expansion that results in an array and make it be used as a
} scalar value in all other cases, independent of quoting

No, that's too confusing and not necessary.  The only real issue is the
"precedence" if you will, of subscripting vs. joining.  We "solved" the
problem of subscripting scalars that were accidentally treated as arrays,
by forcing all quoted arrays to be taken as scalars; I think we need to
back off from that and concentrate on not accidentally treating scalars
as arrays.

I don't think that's going to mess up the "substitution rules" too badly.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

Bart Schaefer wrote:

> Now that I've seen/used it in practice a few times, I'm not so sure any
> more.  The more radical change, and one which bothers me a lot, is the
> effect when the array has more than one element:
> 
> 	foo=(xa yb zc)
> 	print -l "${${(@)foo}[1]}" "${(@)${foo}[1]}"
> 
> Zsh 3.0.5 prints
> 
> 	xa
> 	x
> 
> But 3.1.5-pws-14 prints
> 
> 	x
> 	xa
> 
> That's completely reversed the semantics, and thus is a serious problem.

(Do you really get `xa' in the last case?)

This (and the need to repeat the `(@)') comes from the fact that
multsub() doesn't get any information about where the words came
from. I.e. if they are an `array' (even if only one string) or not. I
already said this at least once. To solve this better we would need a
way to make `paramsubst()' notify `multsub()' if the thing is an
array. This could be done by using a subst.c-global variable that is
set in multsub() and paramsubst() and tested in multsub() after
prefork(), of course.
And then we would have to decide when we want to accept an array at
the call of multsub(). Always, if the sub-expression is an array? Even
if the whole thing is in double quotes? If it is in double quotes and
we don't want it always, when? Hm, maybe when the `(@)' flag is given
or the things came from an array. But then there would be no way to
make an array be treated as a scalar further down up then by using
the `(j:...:)' flag. But then again, this may be ok.

Or maybe we make the inner expression be taken as an array if it is a
parameter expansion that results in an array and make it be used as a
scalar value in all other cases, independent of quoting (which means
that the quoting will only have it's normal effect on the inner
expressions). That way the `(@)' has only an effect to the `outside',
not on the treatment of an inner expression when given in an outer
one (something I already described -- and didn't like, too). This
would give the same as 3.0.5 again for your example above, with the
`x' returned as an array, which noone will notice (and the `(@)'
wouldn't have any effect in `"${(@)${foo}[1]}"').

There may be other problems I don't see, though (apart from changing
the manual again).

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Bart Schaefer]

On Apr 12,  9:17am, Sven Wischnowsky wrote:
} Subject: Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working proper
}
} The rule of thumb is now that one should use the `(@)' flag whenever
} the thing is in double quotes and one wants to work on arrays.

When I first saw the patch that introduced that change, I thought that
introducing the extra consistency sounded like a good thing.

	bar=(xa)
	print -l "${${(@)bar}[1]}" "${(@)${bar}[1]}"

Zsh 3.0.5 prints

	x
	x

But 3.1.5-pws-14 prints

	x
	xa

which is nice, because you can tell a singleton array from a scalar.

Now that I've seen/used it in practice a few times, I'm not so sure any
more.  The more radical change, and one which bothers me a lot, is the
effect when the array has more than one element:

	foo=(xa yb zc)
	print -l "${${(@)foo}[1]}" "${(@)${foo}[1]}"

Zsh 3.0.5 prints

	xa
	x

But 3.1.5-pws-14 prints

	x
	xa

That's completely reversed the semantics, and thus is a serious problem.

Another difficulty is that (@) has to be repeated at *every* level of a
nested expansion, from the innermost ${...} at which an array is desired
all the way to the outermost braces.

	"${${${${(@)foo}}}}"		== "$foo[@]"

To get the equivalent effect now, having to write

	"${(@)${(@)${(@)${(@)foo}}}}"	== "$foo[@]"

is error-prone and just plain annoying.  There must be a better way.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Sven Wischnowsky]

mason@primenet.com.au wrote:

> Here's a fragment of the expansion similar to that used in
> cvsentries() in Misc/compctl-examples 
> 
> % zsh-3.1.5
> % a=("${(f@)$(echo foo; echo bar)}"); print -l $a
> foo
> bar
> % a=("${${(f@)$(echo foo; echo bar)}:#f*}"); print -l $a
> bar
> % zsh-3.1.5-pws-14
> % a=("${(f@)$(echo foo; echo bar)}"); print -l $a
> foo
> bar
> % a=("${${(f@)$(echo foo; echo bar)}:#f*}"); print -l $a
> 
> %

Yes, this has changed lately because the old behavior sometimes
yielded unpredictable results. The rule of thumb is now that one
should use the `(@)' flag whenever the thing is in double quotes and
one wants to work on arrays.

Bye
 Sven

--- ../zold/Misc/compctl-examples	Mon Oct 26 23:59:45 1998
+++ Misc/compctl-examples	Mon Apr 12 09:14:44 1999
@@ -11,15 +11,15 @@
 # page.
 #
 #------------------------------------------------------------------------------
-hosts=("${${(s: :)${(s:	:)${${(f)$(</etc/hosts)}%%\#*}#*[ 	]*}}:#}")
-ports=( "${${${(f)$(</etc/services)}:#\#*}%%[ 	]*}" )
+hosts=("${(@)${(@s: :)${(@s:	:)${(@)${(@f)$(</etc/hosts)}%%\#*}#*[ 	]*}}:#}")
+ports=( "${(@)${(@)${(@f)$(</etc/services)}:#\#*}%%[ 	]*}" )
 
 # groups=( $(cut -d: -f1 /etc/group) )
 # groups=( $(ypcat group.byname | cut -d: -f1) ) # if you use NIS
 
 # It can be done without forking, but it used too much memory in old zsh's:
-groups=( "${${(f)$(</etc/group)}%%:*}" )
-#groups=( "${${(f)$(ypcat groups)}%%:*}" ) # if you use NIS
+groups=( "${(@)${(@f)$(</etc/group)}%%:*}" )
+#groups=( "${(@)${(@f)$(ypcat groups)}%%:*}" ) # if you use NIS
 
 # Completion for zsh builtins.
 compctl -z -P '%' bg
@@ -251,7 +251,7 @@
 # Note that 'r[-exec,;]' must come first
 if [[ -r /proc/filesystems ]]; then
     # Linux
-    filesystems='"${${(f)$(</proc/filesystems)}#*	}"'
+    filesystems='"${(@)${(@f)$(</proc/filesystems)}#*	}"'
 else
     filesystems='ufs 4.2 4.3 nfs tmp mfs S51K S52K'
 fi
@@ -563,7 +563,7 @@
 cvsentries() {
     setopt localoptions nullglob unset
     if [[ -f ${pref}CVS/Entries ]]; then
-	reply=( "${pref}${^${${${(f@)$(<${pref}CVS/Entries)}:#D*}#/}%%/*}" )
+	reply=( "${pref}${^${(@)${(@)${(f@)$(<${pref}CVS/Entries)}:#D*}#/}%%/*}" )
     fi
 }
 
@@ -574,7 +574,7 @@
 }
 
 cvsrevisions() {
-    reply=( "${${${(M)${(f)$(cvs -q status -vl .)}:#	*}##[ 	]##}%%[ 	]*}" )
+    reply=( "${(@)${(@)${(@M)${(@f)$(cvs -q status -vl .)}:#	*}##[ 	]##}%%[ 	]*}" )
 }
 
 cvsrepositories() {
@@ -582,7 +582,7 @@
     [[ -f CVS/Root ]] && root=$(<CVS/Root)
     reply=(
 	$root/^CVSROOT(:t)
-	"${${(M)${(f)$(<$root/CVSROOT/modules)}:#[^#]*}%%[ 	]*}"
+	"${(@)${(@M)${(@f)$(<$root/CVSROOT/modules)}:#[^#]*}%%[ 	]*}"
     )
 }
 
@@ -622,7 +622,7 @@
 	'c[-1,--rcfile]' -f - \
 	'p[1] s[-b]' -k '(p l c i b a)' - \
 	'c[-1,--queryformat] N[-1,{]' \
-		-s '"${${(f)$(rpm --querytags)}#RPMTAG_}"' -S '}' - \
+		-s '"${(@)${(@f)$(rpm --querytags)}#RPMTAG_}"' -S '}' - \
 	'W[1,-q*] C[-1,-([^-]*|)f*]' -f - \
 	'W[1,-i*], W[1,-q*] C[-1,-([^-]*|)p*]' \
 		-/g '*.rpm' + -f -- rpm
@@ -635,7 +635,7 @@
 function talkmatch {
     local u
     reply=($(users))
-    for u in "${${(f)$(rwho 2>/dev/null)}%%:*}"; do
+    for u in "${(@)${(@f)$(rwho 2>/dev/null)}%%:*}"; do
 	reply=($reply ${u%% *}@${u##* })
     done
 }

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

BUG: zsh-3.1.5-pws-14: parameter expansion not working properly

[Geoff Wing]

Here's a fragment of the expansion similar to that used in
cvsentries() in Misc/compctl-examples 

% zsh-3.1.5
% a=("${(f@)$(echo foo; echo bar)}"); print -l $a
foo
bar
% a=("${${(f@)$(echo foo; echo bar)}:#f*}"); print -l $a
bar
% zsh-3.1.5-pws-14
% a=("${(f@)$(echo foo; echo bar)}"); print -l $a
foo
bar
% a=("${${(f@)$(echo foo; echo bar)}:#f*}"); print -l $a

%
-- 
Geoff Wing   <gcw@pobox.com>            Mobile : (Australia) 0412 162 441
Work URL: http://www.primenet.com.au/   Ego URL: http://pobox.com/~gcw/