PATCH: pws-19: minor syntactic innovation

PATCH: pws-19: minor syntactic innovation

[Peter Stephenson]

Just thought of a trivial way to take a lot of grief out of parameter
substitution.  This patch makes the following valid:

  ${${${${(f)"$(typeset)"}:#*local *\=*}%%\=*}##* }

replacing the equivalent but grotesquer

  "${(@)${(@)${(@)${(@f)$(typeset)}:#*local *\=*}%%\=*}##* }"

Since you got nothing when you did that before, I don't see how it can
break anything.  The key point is that quotation of a substitution is
already tested for by looking at whether it begins with a quoted $, not by
whether the parent substitution is quoted.

--- Src/subst.c.qt	Fri May 21 09:14:43 1999
+++ Src/subst.c	Sun May 23 17:07:33 1999
@@ -976,7 +976,9 @@
 		zerr("bad substitution", NULL, 0);
 		return NULL;
 	    }
-	} else
+	} else if (INULL(*s))
+	    s++;
+	else
 	    break;
     }
     globsubst = globsubst && !qt;
@@ -999,6 +1001,8 @@
 	    idbeg = val;
 	copied = 1;
 	*s = sav;
+	while (INULL(*s))
+	    s++;
 	v = (Value) NULL;
     } else if (aspar) {
 	if ((v = getvalue(&s, 1))) {

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: PATCH: pws-19: minor syntactic innovation

[Bart Schaefer]

On May 23,  5:15pm, Peter Stephenson wrote:
} Subject: PATCH: pws-19: minor syntactic innovation
}
} Just thought of a trivial way to take a lot of grief out of parameter
} substitution.  This patch makes the following valid:
} 
}   ${${${${(f)"$(typeset)"}:#*local *\=*}%%\=*}##* }

Hooray, hooray, hooray.  About six times a year I try that kind of syntax
(having forgotten it doesn't work), grumble to myself, and do it some other
way.  Thank you.  I always assumed there was some incredibly hairy parsing
thing that had to be done to consume the quoted string at that point.

} replacing the equivalent but grotesquer
} 
}   "${(@)${(@)${(@)${(@f)$(typeset)}:#*local *\=*}%%\=*}##* }"

Perfect.

I'm tempted to try something similar in 3.0.6 ... except because of this:

} Since you got nothing when you did that before, I don't see how it can
} break anything.  The key point is that quotation of a substitution is
} already tested for by looking at whether it begins with a quoted $, not by
} whether the parent substitution is quoted.

If I put this patch of yours into 3.0.6, will it behave differently than
it does in 3.1.5 (e.g., because of the mult_isarr change)?  If so, I think
I'd rather not introduce differing new behaviors in the two branches.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: pws-19: minor syntactic innovation

[Bart Schaefer]

On May 23,  5:15pm, Peter Stephenson wrote:
} Subject: PATCH: pws-19: minor syntactic innovation
}
} Just thought of a trivial way to take a lot of grief out of parameter
} substitution.  This patch makes the following valid:
} 
}   ${${${${(f)"$(typeset)"}:#*local *\=*}%%\=*}##* }

OK, the patch is straightforward to adapt and the results seem to be the
same in 3.0.6-pre-3 and 3.1.5-pws-20.

A few questions come up, however:

What's the parse of something like ${(f)"${"$(typeset)"}"} ?  It doesn't
seem to be the `obvious' one -- it appears that the quoted strings are
actually nesting, because of the enclosing braces, which is something
pretty radically new.  (I may be wrong.)  Is this a good thing?

What's the right way to document this change, including odd stuff like
the above?

Should the FAQ recommend using this form in some circumstances because
of the (@) change in 3.1.5?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: PATCH: pws-19: minor syntactic innovation

[Andrej Borsenkow]

>
> What's the parse of something like ${(f)"${"$(typeset)"}"} ?  It doesn't
> seem to be the `obvious' one -- it appears that the quoted strings are
> actually nesting, because of the enclosing braces, which is something
> pretty radically new.  (I may be wrong.)  Is this a good thing?
>

No. It breaks the basic shell grammar. The main problem is, it can have
unexpected side effects when running in sh or ksh compatibility mode - and
this is definitely bad thing.

/andrej

Re: PATCH: pws-19: minor syntactic innovation

[Peter Stephenson]

"Andrej Borsenkow" wrote:
> > What's the parse of something like ${(f)"${"$(typeset)"}"} ?  It doesn't
> > seem to be the `obvious' one -- it appears that the quoted strings are
> > actually nesting, because of the enclosing braces, which is something
> > pretty radically new.  (I may be wrong.)  Is this a good thing?
> 
> No. It breaks the basic shell grammar. The main problem is, it can have
> unexpected side effects when running in sh or ksh compatibility mode - and
> this is definitely bad thing.

This isn't due to the change I made, it simply showed up because of that.
The cause is dquote_parse(), which is doing brace-counting; in other words,
it rejects the second `"' as the end of the quoted string because it
detected the ${.  So this behaviour seems to be entirely deliberate.

I'm not so sure it's either new or wrong.  I discovered in an
initialisation file here,
      if [ -n "${BASH_VERSION:-""}" ]; then
which is obviously for bash rather than zsh (and works in bash, too).  If
the quotes don't parse in a nested fashion, this is meaningless, and
wouldn't parse in zsh because it always matches up braces, including inside
quotes.

Furthermore,
  bash$ echo "${FOO"<RET>
  > 

It's at the continuation prompt, so it's still parsing after the brace,
just like zsh (bash will report an error if it ever gets the remaining
`"}"', though).  sh reports `bad subsitution' at this point, however; but
even so, I can't think offhand of a working piece of sh code which would be
broken by this feature.  The fact that bash works happily as sh tends to
reinforce that view.

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

RE: PATCH: pws-19: minor syntactic innovation

[Andrej Borsenkow]

>
> This isn't due to the change I made, it simply showed up because of that.
> The cause is dquote_parse(), which is doing brace-counting; in
> other words,
> it rejects the second `"' as the end of the quoted string because it
> detected the ${.  So this behaviour seems to be entirely deliberate.
>

I beg your pardon, I was just too eager. Looking more closely in Single Unix
reveals:

The dollar-sign retains its special meaning introducing parameter expansion
(see Parameter Expansion ), a form of command substitution (see Command
Substitution ), and arithmetic expansion (see Arithmetic Expansion ). The
input characters within the quoted string that are also enclosed between
"$(" and the matching "(" will not be affected by the double-quotes, but
rather define that command whose output replaces the $(...) when the word is
expanded. The tokenising rules in Token Recognition are applied recursively
to find the matching ")". Within the string of characters from an enclosed
${ to the matching "}", an even number of unescaped double-quotes or
single-quotes, if any, must occur. A preceding backslash character must be
used to escape a literal "{" or "}". The rule in Parameter Expansion will be
used to determine the matching "}".

In other words, Zsh (and bash) both behave "POSIXLY-correct" at least in the
respect to correct expressions.


/andrej

PATCH: pws-19: document minor syntactic innovation

[Peter Stephenson]

"Bart Schaefer" wrote:
> On May 23,  5:15pm, Peter Stephenson wrote:
> }   ${${${${(f)"$(typeset)"}:#*local *\=*}%%\=*}##* }

I should note that things like ${"foo"} only work by accident: you'll find,
for example, that ${"foo"%bar} doesn't work.  That would require swallowing
nulled out quotes at another point in paramsubst(), which I hinted at
before.  I didn't do this because I didn't see a use for it, but maybe it's
more consistent that way?

> What's the right way to document this change, including odd stuff like
> the above?

Here's a possible chunk for 3.1.5-pws-20.

> Should the FAQ recommend using this form in some circumstances because
> of the (@) change in 3.1.5?

Well, the new nested substitution section (3.22) is really about getting a
parameter evaluated as a parameter name and I'm not sure it's a good idea
to put in a lot of stuff about other flags and their uses, which could get
quite involved.  Maybe it needs a separate question, but I'm not sure I can
even remember the full effect of the changes.

--- Doc/Zsh/expn.yo.qt	Mon May 10 10:18:42 1999
+++ Doc/Zsh/expn.yo	Tue Jun  1 11:41:35 1999
@@ -502,6 +502,15 @@
 deleted.  The form with tt($LPAR())...tt(RPAR()) is often useful in
 combination with the flags described next; see the examples below.
 
+Note that double quotes may appear around nested quotations, in which case
+only the part inside is treated as quoted; for example, tt(${(f)"$(foo)"})
+quotes the result of tt($(foo)), but the flag `tt((f))' (see below) is
+applied using the rules for unquoted substitutions.  Note further that
+quotes are themselves nested in this context; for example, in
+tt("${(@f)"$(foo)"}"), there are two sets of quotes, one surrounding the
+whole expression, the other (redundant) surrounding the tt($(foo)) as
+before.
+
 subsect(Parameter Expansion Flags)
 cindex(parameter expansion flags)
 cindex(flags, parameter expansion)

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

Re: PATCH: pws-19: document minor syntactic innovation

[Bart Schaefer]

On Jun 1, 11:56am, Peter Stephenson wrote:
} Subject: PATCH: pws-19: document minor syntactic innovation
}
} I should note that things like ${"foo"} only work by accident: you'll find,
} for example, that ${"foo"%bar} doesn't work.  That would require swallowing
} nulled out quotes at another point in paramsubst(), which I hinted at
} before.  I didn't do this because I didn't see a use for it, but maybe it's
} more consistent that way?

If it's messy, I think it could be left out, but yes, it is more consistent
that way.

} > What's the right way to document this change, including odd stuff like
} > the above?
} 
} Here's a possible chunk for 3.1.5-pws-20.
[...]
} +Note that double quotes may appear around nested quotations, in which case

Perhaps "nested expansions" rather than "nested quotations"?

} > Should the FAQ recommend using this form in some circumstances because
} > of the (@) change in 3.1.5?
} 
} Well, the new nested substitution section (3.22) is really about getting a
} parameter evaluated as a parameter name and I'm not sure it's a good idea
} to put in a lot of stuff about other flags and their uses, which could get
} quite involved.  Maybe it needs a separate question, but I'm not sure I can
} even remember the full effect of the changes.

The full effect of the (@) changes?  It's pretty simple, really.  In 3.0,
once something becomes an array as a result of (@) or (f) or (s//), it is
always an array, even inside quotes, no matter how deeply nested the ${ }
get.  In 3.1.5-pws-XX for XX > 15 (I think it was), array-ness persists
only for the current level of ${ }; if there's another ${ } around that,
quoted-ness takes over again.

So to get certain quoted array expansions to behave the same in 3.0.6 and
3.1.5-pws-20+, you must either put in an (@) at every ${ } (which is ugly
and, in 3.0.6, redundant), or you must `push in the quotes' as deeply into
the nested ${ } as possible, e.g. to immediately surround the ${(@)...}.

For example, in 3.0.6-pre-4 and later, the following all are (will be)
equivalent:

	"${${${(f)$(print -l *)}}}"		(1)
	${"${${(f)$(print -l *)}}"}		(2)
	${${"${(f)$(print -l *)}"}}		(3)
	${${${(f)"$(print -l *)"}}}		(4)

In 3.1.5-pws-20, (1) == (2) [both are strings] and (3) == (4) [both are
arrays], but [obviously] (1) != (4).

Oh, and of course the [@] subscript and subscript number ranges behave
just like the (@) flag as far as this is concerned.

I admit I haven't worked out whether there are other ramifications when
SH_WORD_SPLIT is set .... 

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: PATCH: pws-19: document minor syntactic innovation

[Peter Stephenson]

"Bart Schaefer" wrote:
> On Jun 1, 11:56am, Peter Stephenson wrote:
> } Subject: PATCH: pws-19: document minor syntactic innovation
> }
> } I should note that things like ${"foo"} only work by accident: you'll find,
> } for example, that ${"foo"%bar} doesn't work.  That would require swallowing
> } nulled out quotes at another point in paramsubst(), which I hinted at
> } before.  I didn't do this because I didn't see a use for it, but maybe it's
> } more consistent that way?
> 
> If it's messy, I think it could be left out, but yes, it is more consistent
> that way.

This is the patch below; ${"foo"%bar} will now work.  There are still
probably lots of other possible places, but I don't think it's worth
searching for them; if it seems sensible to allow quotes there, someone
will notice at some point.

> } +Note that double quotes may appear around nested quotations, in which case
> 
> Perhaps "nested expansions" rather than "nested quotations"?

Yes, I'll put this onto the list of minor tweaks I have to do.  I've got
the second description of subscripts (outside a nested quotation) wrong,
though the first (directly onto a parameter name) is now correct.

> The full effect of the (@) changes?  It's pretty simple, really.

Yeah, right.

--- Src/subst.c.inull	Tue Jun  1 09:38:34 1999
+++ Src/subst.c	Wed Jun  2 09:54:59 1999
@@ -1168,6 +1168,8 @@
     }
 
     idend = s;
+    while (INULL(*s))
+	s++;
     if ((colf = *s == ':'))
 	s++;
 
-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

sh compatibility RE: PATCH: pws-19: document minor syntactic innovation

[Andrej Borsenkow]

> 
> This is the patch below; ${"foo"%bar} will now work.  There are still
> probably lots of other possible places, but I don't think it's worth
> searching for them; if it seems sensible to allow quotes there, someone
> will notice at some point.
> 

How compatible with shell is it?

bor@itsrm2:~%> foo=bar
bor@itsrm2:~%> print $"foo"
$foo
bor@itsrm2:~%> print ${"foo"}
bar
bor@itsrm2:~%> print ${"bar":-com}

but

bor@itsrm2:~%> /sbin/xpg4/sh (for what it's worth :-)
$ foo=bar
$ print $"foo"
$foo
$ print ${"foo"}
/sbin/xpg4/sh: ${"foo"}: bad substitution
$ print ${"bar":-com}
/sbin/xpg4/sh: ${"bar":-com}: bad substitution

Re: sh compatibility RE: PATCH: pws-19: document minor syntactic innovation

[Peter Stephenson]

"Andrej Borsenkow" wrote:
> > This is the patch below; ${"foo"%bar} will now work.
> 
> How compatible with shell is it?
> 
> bor@itsrm2:~%> /sbin/xpg4/sh (for what it's worth :-)
> $ foo=bar
> $ print ${"foo"}
> /sbin/xpg4/sh: ${"foo"}: bad substitution
> $ print ${"bar":-com}
> /sbin/xpg4/sh: ${"bar":-com}: bad substitution

Personally, I'm not too worried about things which give `bad substitution'
in other shells --- note that the presence of quotes never gave you that
message in zsh before, it simply didn't do anything.  We've already got a
whole heap of cases which zsh accepts but other shells don't.  If we embark
on a syntax-tightening exercise, I rather suspect we'll never get to the
end.

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy