PATCH: more substitution documentation

PATCH: more substitution documentation

[Peter Stephenson]

This is more less my explanation of Andrej's other query yesterday, which I
think I now understand.  If anyone wants to suggest rearranging the
information in this part of the manual, please go ahead and say where you
want it to appear.

--- Doc/Zsh/expn.yo.js	Wed Mar 17 11:51:19 1999
+++ Doc/Zsh/expn.yo	Thu Mar 18 10:11:47 1999
@@ -729,6 +729,21 @@
 )
 enditem()
 
+Any joining and splitting of words which is necessary occurs in that order,
+and after any other substitutions performed on the value at that level of
+substitution; this includes implicit splitting on the characters in
+tt($IFS) when the option tt(SH_WORD_SPLIT) is set.  In particular, when
+splitting is requested on an array value it is first joined, either using
+any string given by the tt(LPAR()j)tt(RPAR()) flag, or a space if there is
+none.  So if tt($foo) contains the array tt(LPAR()ax1 bx1)tt(RPAR()), then
+tt(${(s/x/)foo}) produces the words `tt(a)', `tt(1 b)' and `tt(1)', while
+tt(${(j/x/s/x/)foo}) produces `tt(a)', `tt(1)', `tt(b)' and `tt(1)'.  As
+substituion occurs before either joining or splitting, the operation
+tt(${(s/x/)foo%%1*}) first generates the modified array tt(LPAR()ax
+bx)tt(RPAR()), which is joined to give tt("ax bx"), and then split to give
+`tt(a)', `tt( b)' and `'.  The final empty string will then be elided, as
+it is not in double quotes.
+
 texinode(Command Substitution)(Arithmetic Expansion)(Parameter Expansion)(Expansion)
 sect(Command Substitution)
 cindex(command substitution)

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Final touch? RE: PATCH: more substitution documentation

[Andrej Borsenkow]

I think, there is a couple more things that need to be explicitly stated.

1. Nested substitution:

1.1 substitution is done reqursively, including implicit word splitting if
SH_WORD_SPLIT is set. This may be obvious, but it is not so in manual. (I
believed for a long time, that field splitting is done once after all is
done)

1.2 The result of substitution is a *list* (possibly empty or with having
only one element).

1.3 Implicit joinig in quotes happens by *outer* subst when it is going to
use the value; not by *inner* subst after the value is computed. Again, I
always believed, (@) prevents *outer* level from joining words (hence all
this confusion). That explains (for me):

itsrm2% foo=(a b c d)
itsrm2% args "${(@)foo[2,3]}"
2
b
c

First the foo[2,3] is taken. Because of (@) it is not joined. This is still
vague, as we actually have two lists here - $foo and $foo[2,3]. It seems,
that foo[2,3] is treated as a single entity. It probably needs to be
explained too. That is, the difference betwee foo[...] and ${...}[...].

itsrm2% args "${${(@)foo}[2,3]}"
1
 b

Outer substitution first joins the list.

itsrm2% args "${(@)${foo}[2,3]}"
0

Inner subst does join the list foo (as it is in quotes) - outer one does
*not* join it - and we get a list with one word. Rules 1.2 and 1.3.

2. (@) flag

In double quotes, it prevents the words of list that results from inner
subst to be joined and converted to scalar, even if list has only single
word (this is very important)!! That is, even if inner subst results in one
word, it still treated as list (array) with one word.

3. Splitting

To my great suprise I found, that splitting works even in double quotes!!!

itsrm2% args "${(f)$(print "a\nb\nc")}"
3
a
b
c
itsrm2% foo=axbxc
itsrm2% args "${(s/x/)foo}"
3
a
b
c
foo="a b c"
itsrm2% args "$=foo"
3
a
b
c

That is, no (@) is needed in first case (that is also logical, if we take
the meaning of (@) as in 2). Again, it should be mentioned explicitly. This
is also logical, if we assume, that joinig occurs in outer subst - as here
we have no outer subst at all :-)

4. $= meaning

As you see, the spplitting, that is done by $= is *not* the same as default
field splitting with SH_WORD_SPLIT. The latter does not happen in double
quotes - the former does. I also think it is O.K. - but then, the wording in
manual must be changed. Not "turns on SH_WORD_SPLIT" - but "performs word
splitting using the same rules as SH_WORD_SPLIT".

Looking back, this makes the whole story probably predictable :-) this also
explains difference between "${foo[1]}" and "${${(@)foo}[1]}". foo[1] is
treated as single entity and results in first elemnt of foo. In the latter
case, first value of foo is taken (list), that is then passed as list ((@)
flag) to upper subst; that joins the list together (no (@)) and treats it as
scalar. Nice. To make the outer treat list as list (array) you need te
second (@) - exactly as Sven said.

So, the whole for a ${(flags)var-mods} looks like

 1. take the value of var. The result is list (array) of words (may be empty
or having only one element)
 2. If this is quoted and no (@) is present, join the list with the first
character if IFS (I think, it is correct? Not always with space?) thus
producing a single word (again, actually a list with single word)
 3. apply mods separately for every word in list
 4. join the words again if (j) is present
 5. split using (s) or $= if present
 6. If no $= or (s) is present and the whole is not quoted, apply
SH_WORD_SPLIT:

itsrm2% foo="x y:a b"
itsrm2% setopt shwordsplit
itsrm2% args ${(s/:/)foo}
2
x y
a b
itsrm2% args ${foo}
3
x
y:a
b

 7. the result is again a (possibly empty) list of words. If the whole is
not quoted, the empty words are removed

 The steps 1-7 are done recursively for every nested substitution (I hate
this word - spelling, that is :-)

And a little comment above about subscription.

cheers

/andrej