Re: Tutorial on zsh substitutions (Re: OPTARG not being set?)

Re: Tutorial on zsh substitutions (Re: OPTARG not being set?)

[Bart Schaefer]

On Feb 3,  8:16pm, Sweth Chandramouli wrote:
> Subject: Re: Tutorial on zsh substitutions (Re: OPTARG not being set?)
> On Wed, Feb 03, 1999 at 12:56:50PM -0800, Bart Schaefer wrote:
> 	all of that said, it would be a pretty big task to reorganize all
> of the docs that way; if there aren't any objections, i would be willing
> to at least give it a shot.

You'd need to get the pws-6 release plus the four or five doc changes that
have been posted since.  You need to get the yodl suite and learn the yodl
language.  And then you'd need to cope with the ongoing changes to zsh; the
pace of development right now is the fastest it has been in several years.

In short, I don't think anybody would object, but this might not be the best
time to do it (except maybe for the learning yodl part).

> 	even though i don't speak c, i think i might get a better idea of
> what is going on if i could see what is being parsed when in the code 
> itself.  could you (or someone else) give me a pointer to which file, and
> where in that file, this expansion is processed?

It's the function singsub() in the file Src/subst.c; nested substitutions
are handled by a call to multsub().  Good luck understanding it.

> 	should this type of discussion be on zsh-workers instead?

No; it's just that the volume of traffic on zsh-workers has been very high
lately, and the list forwarding arrangement at the new host site makes it
less obvious that the list originated on zsh-users when delivering to those
of us who get it through zsh-workers.

> > > i originally thought that this meant that ${(e)${bob}} would parse to
> > > ${(e)joe}, and then perform parameter expansion on ${joe}
> > 
> > No.  "The result" is just "joe".  Outside in, not inside out -- the outer
> > ${...} is already "used up" by the time the (e) is applied.  Get it?
> 	not really.  as i said above, i see what you are saying, but the
> way i interpret it, that would mean that either the outer ${...} was turned
> into a no-op

That pretty much is what happens.  The outer ${...} "queues up" all the
flag and modifier manipulations that need to happen and then gives control
over to the inner ${...} or $(...).  When that finally comes back, all the
flags and modifiers are applied, with (e) always applied last of all, and
then the resulting text is returned.

Re: Tutorial on zsh substitutions (Re: OPTARG not being set?)

[Sweth Chandramouli]

On Wed, Feb 03, 1999 at 12:56:50PM -0800, Bart Schaefer wrote:
> It's true that "the value of NAME" is a bad turn of phrase there.  A better
> way to describe it would be to say
> 
>     ${THING}
> 	When THING is a parameter name, the value of that parameter
> 	replaces THING.  When THING is another substitution expression,
> 	introduced by a leading '$', the value of that expression
> 	replaces THING.  All other processing specified by parameter
> 	substitution flags and modifiers is applied to the replacment,
> 	and the result is substituted for the entire expression.
> 
> The reason it's not like that is because (a) the manual was incrementally
> built up from documentation that predates nested substitutions, and (b)
> there was a desire not to "forward reference" flags, modifiers, and nested
> expression substitution until the simpler parameter-name-only cases were
> fully described.

	i think that these sorts of things could be explained by building
up from the simplest situations to the most complex, but that it would be
a lot easier if the concepts used in explaining them were built up in the
same way.  in the current docs, for example, the paramaters section comes
after the section on parameter expansion; it would seem to make more sense
to me to explain what a parameter is first, and set out the distinction
between names and values there, so that the terms can be more clearly used
when discussing expansion later on.
	this would, as you say, require people understanding a basic shell
grammar, but i notice that the info from the beginning of the zshmisc
manpage has been grouped together at the start of pws' online manpage 
under the heading "shell grammar".  while i have seen some people argue
that the manpages should be purely reference, without examples and such
things, i can't see how even those people would object to having the
concepts in the manpage build on one another, so that a person reading
through it for the first time would be able to follow what was going on
more easily.
	all of that said, it would be a pretty big task to reorganize all
of the docs that way; if there aren't any objections, i would be willing
to at least give it a shot.


[snip]
> Not exactly.  The difference is essentially that a name can never begin
> with a '$' character, so anything not beginning with '$' is a name that
> is used to look up a value, and anything that does begin with '$' is an
> expression that must be recursively interpreted to produce a value.
[snip]
> It is standard; what you're proposing amounts to an order-of-evaluation
> change.  That is, presently nested ${...} expressions are evaluated from
> the outside in, whereas you want them evaluated from the inside out.
	i see what you are saying here, but don't think that that explains
the actual behaviour.  my problem, i think, is that i am only able to discuss
it on a theoretical level based on the logic of operation association and
the effects that i observe; it may well be that what you describe _is_ what
is happening, but as it is, i don't see it.
	even though i don't speak c, i think i might get a better idea of
what is going on if i could see what is being parsed when in the code 
itself.  could you (or someone else) give me a pointer to which file, and
where in that file, this expansion is processed?

> > > What Sven's new (P) flag does is force the value to be interpreted as a
> > > new name, the way the syntax misled you expect all along.  (Maybe the
> > > (P) flag ought to be ($), as in ${($)${bob}}.  Or more likely not.)
> > 	is this added in a dev version?  i can't find reference to it
> > in the 3.1.5 docs.
> 
> It was a patch just posted yesterday.  I didn't notice that this message
> was going to zsh-users, where some people wouldn't recognize the context.
	should this type of discussion be on zsh-workers instead?  i posted
it to zsh-users originally because i'm not on the workers list (since, as
i said above, i don't speak c and thus can't be much of a worker).
	(then again, considering that this is the second straight time you
and i have gotten bogged down in differences of opinion on how things in
the docs should be interpreted, maybe it isn't worthy of zsh-workers, either.)

> > > First, note that ${bob} substitutes to "joe".  ${(e)bob} is similar (but
> > > not identical) to $(echo ${bob}).  
> > 	how so?  according to the docs, (e) says to "[p]erform parameter
> > expansion, command substitution, and arithmetic expansion on the result."
> 
> Yes.  What that means is:
>     parameter expansion -- if the result looks like $param or ${...},
> 			   then substitute the value of that expression
>     command substitution -- if the result looks like $(command ...),
> 			    then evaluate command and substitute its output
>     arithmetic expansion -- if the result looks like $[expr], then
> 			    compute expr and substitute that result
> 
> That's "similar (but not identical) to" $(echo) because there really isn't
> any other syntax that expresses the same semantics -- the $(echo) is just
> as close as I could get to another way to explain it.
> 
> > i originally thought that this meant that ${(e)${bob}} would parse to
> > ${(e)joe}, and then perform parameter expansion on ${joe}
> 
> No.  "The result" is just "joe".  Outside in, not inside out -- the outer
> ${...} is already "used up" by the time the (e) is applied.  Get it?
	not really.  as i said above, i see what you are saying, but the
way i interpret it, that would mean that either the outer ${...} was turned
into a no-op, or that it somehow successfully expanded the string '${bob}'
(as opposed to the paramater bob).  i think i need to take a look at the
actual code to figure this out.

> > 	what is the difference between grave-accent command substitution,
> > $(...) command substitution, and eval's pseudo-command substitution?  it
> > looks like grave accents strip meta-char escapes, do param expansion, and 
> > then evaluate, while $(...) just does param expansion and then evaluates, 
> > and eval does param expansion, strips meta-char escapes, and then evaluates.
> 
> The main difference between "eval" and either of $(...) or `...` is that
> eval does not capture the output of the command that gets executed.
> 
> The main difference between `...` and either of $(...) or eval is that
> `...` is a quoting mechansism and gets parsed as one.  This causes the
> stripping of the backslash-escapes, and delays interpretation of other
> quoting mechanisms until the time of the evaluation.

	okay; this, at least, i get.

	thanks,
	sweth.

-- 
Sweth Chandramouli
IS Coordinator, The George Washington University
<sweth@gwu.edu> / (202) 994 - 8521 (V) / (202) 994 - 0458 (F)
<a href="http://astaroth.nit.gwu.edu/~sweth/disc.html">*</a>

Tutorial on zsh substitutions (Re: OPTARG not being set?)

[Bart Schaefer]

On Feb 3,  1:35pm, Sweth Chandramouli wrote:
> Subject: Re: OPTARG not being set?
> On Wed, Feb 03, 1999 at 12:03:02AM -0800, Bart Schaefer wrote:
> > Nope.  Nested ${...} constructs don't act like that in zsh; the doc says
> > 
> > ${NAME}
> >     The value, if any, of the parameter NAME is substituted. [...]
> > 
> >     If a `${'...`}' type parameter expression or a `$('...`)' type command
> >     substitution is used in place of NAME above, it is expanded first and
> >     the result is used as if it were the value of NAME.
> >     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 	
> 	see below for my comments on the syntactic definition of a "name".

It's true that "the value of NAME" is a bad turn of phrase there.  A better
way to describe it would be to say

    ${THING}
	When THING is a parameter name, the value of that parameter
	replaces THING.  When THING is another substitution expression,
	introduced by a leading '$', the value of that expression
	replaces THING.  All other processing specified by parameter
	substitution flags and modifiers is applied to the replacment,
	and the result is substituted for the entire expression.

The reason it's not like that is because (a) the manual was incrementally
built up from documentation that predates nested substitutions, and (b)
there was a desire not to "forward reference" flags, modifiers, and nested
expression substitution until the simpler parameter-name-only cases were
fully described.

> > All the rest of the processing
> > specified by flags or modifiers in the outer braces is applied to the
> > resulting value string or array.  That's why ${(t)${bob}} doesn't work,
> > because a value doesn't have a type; only names have types (which are
> > instructions for interpreting the value).
> 	is the difference between a name and a value semantic instead of
> syntactic, then?

Not exactly.  The difference is essentially that a name can never begin
with a '$' character, so anything not beginning with '$' is a name that
is used to look up a value, and anything that does begin with '$' is an
expression that must be recursively interpreted to produce a value.

Some of the flags then mean, "iff you have a name, change the way you
look it up so that instead of its value you get the (type,length,etc.)
of its value."

> i would assume that the shell decides what is a value
> and what is a name by the context in which it is placed; based on the
> definitions given in the manpage, it seems that ${...} constructs 
> definitionally interpret the first token inside the braces to be a name.

That's because the man page is referring only to parameter substitution at
that point, and parameters have names.  It's explaining the simple case
very simply, at the expense of making the complex case appear even more
complex than it is.

> after reading over the paragraph you quote above a few times, i can see
> how it can be interpreted to mean what you say it means, which is that
> nested uses of ${...} override the syntactic identification of the
> first token of all but the innermost ${...} as a name

The problem with thinking of it that way is that the innermost thing may
not be a name at all.  It could be that a $(command) is innermost.  Such
an inner expression doesn't have a "name" to identify.

> 	2) after i sent in my original question, i did a search on the
> zsh-workers archive, and noticed that you had done some work on this
> same topic, at least in standardizing use of terms like "expansion"
> and "substitution".  i think that it would help a lot to do the same
> thing for terms like "name", "word", and value"

Yes, and "parameter" vs. "variable" too, and all sorts of other things.
The problem with doing so is that it makes various sections of the manual
incomprehensible without having previously understood the "shell grammar"
or some other definition section.

> then i think that this could just be explained in the definitions of
> the various ${...} constructs themselves, rather than in a footnote
> at the end.

As I noted, that organization is intentional, to avoid confusing those
who only care about the simplest case.

> 	3) i'm not a huge fan of "magic" values and exceptions (which is
> one of the reasons that i've never really gotten that into perl); the
> third solution, then, would be to make the behaviour of ${...} standard

It is standard; what you're proposing amounts to an order-of-evaluation
change.  That is, presently nested ${...} expressions are evaluated from
the outside in, whereas you want them evaluated from the inside out.

> > What Sven's new (P) flag does is force the value to be interpreted as a
> > new name, the way the syntax misled you expect all along.  (Maybe the
> > (P) flag ought to be ($), as in ${($)${bob}}.  Or more likely not.)
> 	is this added in a dev version?  i can't find reference to it
> in the 3.1.5 docs.

It was a patch just posted yesterday.  I didn't notice that this message
was going to zsh-users, where some people wouldn't recognize the context.

The HTML manual available at www.zsh.org is now for 3.1.5-pws-5, which is
an unofficial patched version that Peter Stevenson has been releasing.
It has most of the details (including many man page clarifications) up
to but not including the (P) flag.

> > First, note that ${bob} substitutes to "joe".  ${(e)bob} is similar (but
> > not identical) to $(echo ${bob}).  
> 	how so?  according to the docs, (e) says to "[p]erform parameter
> expansion, command substitution, and arithmetic expansion on the result."

Yes.  What that means is:
    parameter expansion -- if the result looks like $param or ${...},
			   then substitute the value of that expression
    command substitution -- if the result looks like $(command ...),
			    then evaluate command and substitute its output
    arithmetic expansion -- if the result looks like $[expr], then
			    compute expr and substitute that result

That's "similar (but not identical) to" $(echo) because there really isn't
any other syntax that expresses the same semantics -- the $(echo) is just
as close as I could get to another way to explain it.

> i originally thought that this meant that ${(e)${bob}} would parse to
> ${(e)joe}, and then perform parameter expansion on ${joe}

No.  "The result" is just "joe".  Outside in, not inside out -- the outer
${...} is already "used up" by the time the (e) is applied.  Get it?

> > 	bob='$joe'		# Note ${bob} now substitutes $joe
> > 	${(e)${bob}}		# Like $(echo $joe) --> hello world
> 
> 	to do this dynamically (i.e. to get this effect when you don't
> know ahead of time that joe is name of the parameter to be expanded),
> however, you need to do something like bob="${${param_whose_value_is_joe}}",
> which is the original situation that caused me to start trying to figure
> all of this out.

Right.  This is what the new (P) flag is for.  What you need without (P) is

	eval bob=\$${param_whose_value_is_joe}

Note that I *didn't* write \${${param_whose_value_is_joe}}, to make it clear
that the intent is that ${param_whose_value_is_joe} produce a string that
can be interpreted as a name (and NOT produce another '$'-expression).

> 	what is the difference between grave-accent command substitution,
> $(...) command substitution, and eval's pseudo-command substitution?  it
> looks like grave accents strip meta-char escapes, do param expansion, and 
> then evaluate, while $(...) just does param expansion and then evaluates, 
> and eval does param expansion, strips meta-char escapes, and then evaluates.

The main difference between "eval" and either of $(...) or `...` is that
eval does not capture the output of the command that gets executed.

The main difference between `...` and either of $(...) or eval is that
`...` is a quoting mechansism and gets parsed as one.  This causes the
stripping of the backslash-escapes, and delays interpretation of other
quoting mechanisms until the time of the evaluation.

You can see this most obviosly by using an unbalanced double-quote in
the input somewhere and looking at the secondary prompt:

zsh% eval echo This " is unbalanced
dquote> 

zsh% $(echo This " is unbalanced)
cmdsubst> 

zsh% `echo This " is unbalanced`
zsh: unmatched "
zsh: parse error in command substitution