[PATCH] Enable further expansion of parameter name by ${!...}

[PATCH] Enable further expansion of parameter name by ${!...}

[Tomoki Sekiyama]

With this change, ${!...} will enable further expansion of parameter name,
which is equivalent to (P) expansion flag. This will enable zsh to run some
scripts using variable references for bash. For example:

 function trueorfalse {
     local default=$1
     local literal=$2
     local testval=${!literal}

     [[ -z "$testval" ]] && { echo "$default"; return; }
     [[ "0 no No NO false False FALSE" =~ "$testval" ]] && { echo "False"; return; }
     [[ "1 yes Yes YES true True TRUE" =~ "$testval" ]] && { echo "True"; return; }
     echo "$default"
 }

 echo $(trueorfalse 0 SOME_VAR)
---
 Doc/Zsh/expn.yo | 5 +++++
 Src/subst.c     | 4 ++++
 2 files changed, 9 insertions(+)

diff --git a/Doc/Zsh/expn.yo b/Doc/Zsh/expn.yo
index 8728803..1029aef 100644
--- a/Doc/Zsh/expn.yo
+++ b/Doc/Zsh/expn.yo
@@ -833,6 +833,11 @@ the pattern tt(*.c), which may be expanded by filename generation, but
 tt(${${~foo}//\*/*.c}) substitutes to the string tt(*.c), which will not
 be further expanded.
 )
+item(tt(${!)var(spec)tt(}))(
+This is equivalent to 'tt((P))' flag, which forces the value of the
+parameter var(name) to be interpreted as a further parameter name,
+whose value will be used where appropriate.
+)
 enditem()
 
 If a tt(${)...tt(}) type parameter expression or a
diff --git a/Src/subst.c b/Src/subst.c
index a2bb648..8897350 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -2179,6 +2179,10 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags)
 		zerr("bad substitution");
 		return NULL;
 	    }
+	} else if (c == '!') {
+	    /* equivalent to (P) */
+	    aspar = 1;
+	    s++;
 	} else if (inbrace && inull(*s)) {
 	    /*
 	     * Handles things like ${(f)"$(<file)"} by skipping 
-- 
1.9.3 (Apple Git-50)

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Peter Stephenson]

On Sun, 25 Jan 2015 04:15:38 -0500
Tomoki Sekiyama <tomoki.sekiyama@gmail.com> wrote:
> With this change, ${!...} will enable further expansion of parameter name,
> which is equivalent to (P) expansion flag. This will enable zsh to run some
> scripts using variable references for bash.

Thanks, this is a useeful idea, however it looks like a bit more work
will be necessary to avoid this being confused with the parameter called
"!", i.e. the last programme run in the background.  Both $! and ${!}
have that meaning.

pws

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Mikael Magnusson]

On Sun, Jan 25, 2015 at 7:14 PM, Peter Stephenson
<p.w.stephenson@ntlworld.com> wrote:
> On Sun, 25 Jan 2015 04:15:38 -0500
> Tomoki Sekiyama <tomoki.sekiyama@gmail.com> wrote:
>> With this change, ${!...} will enable further expansion of parameter name,
>> which is equivalent to (P) expansion flag. This will enable zsh to run some
>> scripts using variable references for bash.
>
> Thanks, this is a useeful idea, however it looks like a bit more work
> will be necessary to avoid this being confused with the parameter called
> "!", i.e. the last programme run in the background.  Both $! and ${!}
> have that meaning.

IIRC, in bash, you can also use ${!foo} even with banghist active. We
probably don't want to replicate that feature, but make sure ${\!foo}
works I suppose?

-- 
Mikael Magnusson

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Bart Schaefer]

On Jan 25,  4:15am, Tomoki Sekiyama wrote:
} Subject: [PATCH] Enable further expansion of parameter name by ${!...}
}
} With this change, ${!...} will enable further expansion of parameter name,
} which is equivalent to (P) expansion flag. This will enable zsh to run some
} scripts using variable references for bash.

Don't we already have something like this for ksh emulation?  Except that
it acts like (k) instead of like (P)?  Although I can't find this in the
documentation anywhere, right at the moment.

I'm not going to commit this without some more discussion, though, because
it seems a bit iffy to have exactly the same syntax mean two such very
different things.  At the very least perhaps both branches should depend
on emulation modes rather than having the bash behavior become a default.

diff --git a/Src/subst.c b/Src/subst.c
index a2bb648..882e62c 100644
--- a/Src/subst.c
+++ b/Src/subst.c
@@ -1734,8 +1734,11 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags)
 	 * doesn't have parameter flags it might be neater to
 	 * handle this with the ^, =, ~ stuff, below.
 	 */
-	if ((c = *s) == '!' && s[1] != Outbrace && EMULATION(EMULATE_KSH)) {
-	    hkeys = SCANPM_WANTKEYS;
+	if ((c = *s) == '!' && s[1] != Outbrace) {
+	    if (EMULATION(EMULATE_KSH))
+		hkeys = SCANPM_WANTKEYS;
+	    else	/* emulate bash equivalent to our (P) */
+		aspar = 1;
 	    s++;
 	} else if (c == '(' || c == Inpar) {
 	    char *t, sav;

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[ZyX]

25.01.2015, 23:25, "Bart Schaefer" <schaefer@brasslantern.com>:
> On Jan 25,  4:15am, Tomoki Sekiyama wrote:
> } Subject: [PATCH] Enable further expansion of parameter name by ${!...}
> }
> } With this change, ${!...} will enable further expansion of parameter name,
> } which is equivalent to (P) expansion flag. This will enable zsh to run some
> } scripts using variable references for bash.
>
> Don't we already have something like this for ksh emulation?  Except that
> it acts like (k) instead of like (P)?  Although I can't find this in the
> documentation anywhere, right at the moment.

I have ksh installed and here `${!VAR}` outputs `VAR` regardless of whether $VAR is defined and what type and what value does it have. [Manual page][1] says that it should actually expand to “the name of the variable referred to by vname” and says that this will be `VAR` unless it is a name reference (I have only tried strings, arrays and associative arrays).

[1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?ksh+1

>
> I'm not going to commit this without some more discussion, though, because
> it seems a bit iffy to have exactly the same syntax mean two such very
> different things.  At the very least perhaps both branches should depend
> on emulation modes rather than having the bash behavior become a default.
>
> diff --git a/Src/subst.c b/Src/subst.c
> index a2bb648..882e62c 100644
> --- a/Src/subst.c
> +++ b/Src/subst.c
> @@ -1734,8 +1734,11 @@ paramsubst(LinkList l, LinkNode n, char **str, int qt, int pf_flags)
>           * doesn't have parameter flags it might be neater to
>           * handle this with the ^, =, ~ stuff, below.
>           */
> - if ((c = *s) == '!' && s[1] != Outbrace && EMULATION(EMULATE_KSH)) {
> -    hkeys = SCANPM_WANTKEYS;
> + if ((c = *s) == '!' && s[1] != Outbrace) {
> +    if (EMULATION(EMULATE_KSH))
> + hkeys = SCANPM_WANTKEYS;
> +    else /* emulate bash equivalent to our (P) */
> + aspar = 1;
>              s++;
>          } else if (c == '(' || c == Inpar) {
>              char *t, sav;

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Bart Schaefer]

On Jan 25, 11:38pm, ZyX wrote:
} Subject: Re: [PATCH] Enable further expansion of parameter name by ${!...}
}
} 25.01.2015, 23:25, "Bart Schaefer" <schaefer@brasslantern.com>:
} >
} > Don't we already have something like this for ksh emulation?  Except that
} > it acts like (k) instead of like (P)?  Although I can't find this in the
} > documentation anywhere, right at the moment.
} 
} I have ksh installed and here `${!VAR}` outputs `VAR` regardless of
} whether $VAR is defined and what type and what value does it have.

I thought I recalled something about that.  So it would appear that
whatever interpretation we're giving it already in ksh mode, is wrong.

} [Manual page][1] says that it should actually expand to "the name of
} the variable referred to by vname" and says that this will be `VAR`
} unless it is a name reference (I have only tried strings, arrays and
} associative arrays).
}
} [1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?ksh+1

So it would appear that ${!var} in bash is ${(P)var} (just tried it, it
does so) where as ${!var} in ksh inhibits the expansion of a nameref
to its value and instead just prints the name to which the nameref
refers, which in zsh is [usually] just ${var}.

Interesting thing I didn't know about namerefs -- if you do

nameref foo=bar
nameref bar=baz

then the nameref for $foo is broken (${!foo} prints ".deleted"), but
if you do it the other way round

nameref bar=baz
nameref foo=bar

then ${!foo} prints "baz" and ${foo} gives the value of ${baz}.

-- 
Barton E. Schaefer

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Tomoki Sekiyama]

Hi all,
Thank you for your comments.

> IIRC, in bash, you can also use ${!foo} even with banghist active. We
> probably don't want to replicate that feature, but make sure ${\!foo}
> works I suppose?

Right, I don't think we want to change banghist behavior when it is active.
We can avoid banghist by ${\!foo} when it is active.

2015-01-25 16:39 GMT-05:00 Bart Schaefer <schaefer@brasslantern.com>:
>
> On Jan 25, 11:38pm, ZyX wrote:
> } Subject: Re: [PATCH] Enable further expansion of parameter name by ${!...}
> }
> } 25.01.2015, 23:25, "Bart Schaefer" <schaefer@brasslantern.com>:
> } >
> } > Don't we already have something like this for ksh emulation?  Except that
> } > it acts like (k) instead of like (P)?  Although I can't find this in the
> } > documentation anywhere, right at the moment.
> }
> } I have ksh installed and here `${!VAR}` outputs `VAR` regardless of
> } whether $VAR is defined and what type and what value does it have.
>
> I thought I recalled something about that.  So it would appear that
> whatever interpretation we're giving it already in ksh mode, is wrong.
>
> } [Manual page][1] says that it should actually expand to "the name of
> } the variable referred to by vname" and says that this will be `VAR`
> } unless it is a name reference (I have only tried strings, arrays and
> } associative arrays).
> }
> } [1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?ksh+1
>

I have tried some patterns with ksh and bash.

### Variable reference
aaa=bbb
bbb=ccc
                    # ksh       bash     zsh(default)       zsh(KSH)
echo $aaa           # bbb       bbb      bbb                bbb
echo ${!aaa}        # aaa       ccc      bad substitution   bbb

### Assoc
typeset -A foo
foo[bar]=baz
foo["baz"]="qux"
baz=piyo
                    # ksh       bash     zsh(default)       zsh(KSH)
echo $foo           #                    baz qux
echo ${foo[bar]}    # baz       baz      baz                baz
echo ${foo["bar"]}  # baz       baz
echo ${foo[baz]}    # qux       qux
echo ${foo["baz"]}  # qux       qux      qux                qux
echo ${!foo[bar]}   # foo[bar]  piyo     bad substitution   bar
echo ${!foo["bar"]} # foo[bar]  piyo     bad substitution
echo ${!foo[@]}     # bar baz   bar baz  bad substitution  "baz" bar

It seems that ${!...} in KSH emulation mode of zsh is only useful for the case
of "${!foo[@]}", which is substituted with the list of keys of the assoc,
that has the same meaning also in bash.
(Although zsh regards baz and "baz", or even 'baz' as different keys...)

According to this, introducing bash-like behaviors instead of "bad
substitution" error will
not hurt zsh even in ksh emulation mode.

Any comments are appreciated.

Thanks,
Tomoki

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[ZyX]

26.01.2015, 23:01, "Tomoki Sekiyama" <tomoki.sekiyama@gmail.com>:
> Hi all,
> Thank you for your comments.
>>  IIRC, in bash, you can also use ${!foo} even with banghist active. We
>>  probably don't want to replicate that feature, but make sure ${\!foo}
>>  works I suppose?
>
> Right, I don't think we want to change banghist behavior when it is active.
> We can avoid banghist by ${\!foo} when it is active.
>
> 2015-01-25 16:39 GMT-05:00 Bart Schaefer <schaefer@brasslantern.com>:
>>  On Jan 25, 11:38pm, ZyX wrote:
>>  } Subject: Re: [PATCH] Enable further expansion of parameter name by ${!...}
>>  }
>>  } 25.01.2015, 23:25, "Bart Schaefer" <schaefer@brasslantern.com>:
>>  } >
>>  } > Don't we already have something like this for ksh emulation?  Except that
>>  } > it acts like (k) instead of like (P)?  Although I can't find this in the
>>  } > documentation anywhere, right at the moment.
>>  }
>>  } I have ksh installed and here `${!VAR}` outputs `VAR` regardless of
>>  } whether $VAR is defined and what type and what value does it have.
>>
>>  I thought I recalled something about that.  So it would appear that
>>  whatever interpretation we're giving it already in ksh mode, is wrong.
>>
>>  } [Manual page][1] says that it should actually expand to "the name of
>>  } the variable referred to by vname" and says that this will be `VAR`
>>  } unless it is a name reference (I have only tried strings, arrays and
>>  } associative arrays).
>>  }
>>  } [1]: http://unixhelp.ed.ac.uk/CGI/man-cgi?ksh+1
>
> I have tried some patterns with ksh and bash.
>
> ### Variable reference
> aaa=bbb
> bbb=ccc
>                     # ksh       bash     zsh(default)       zsh(KSH)
> echo $aaa           # bbb       bbb      bbb                bbb
> echo ${!aaa}        # aaa       ccc      bad substitution   bbb
>
> ### Assoc
> typeset -A foo
> foo[bar]=baz
> foo["baz"]="qux"
> baz=piyo
>                     # ksh       bash     zsh(default)       zsh(KSH)
> echo $foo           #                    baz qux
> echo ${foo[bar]}    # baz       baz      baz                baz
> echo ${foo["bar"]}  # baz       baz
> echo ${foo[baz]}    # qux       qux
> echo ${foo["baz"]}  # qux       qux      qux                qux
> echo ${!foo[bar]}   # foo[bar]  piyo     bad substitution   bar
> echo ${!foo["bar"]} # foo[bar]  piyo     bad substitution
> echo ${!foo[@]}     # bar baz   bar baz  bad substitution  "baz" bar
>
> It seems that ${!...} in KSH emulation mode of zsh is only useful for the case
> of "${!foo[@]}", which is substituted with the list of keys of the assoc,
> that has the same meaning also in bash.
> (Although zsh regards baz and "baz", or even 'baz' as different keys...)

Quote removal is not done in subscripts. In bash you may have `baz`, `'baz'` and `"baz"` and due to quote removal it computes to just `baz`. This is explicitly mentioned in the seventh paragraph of Subscript Parsing subsection of ARRAY PARAMETERS section of `man zshparam`.

>
> According to this, introducing bash-like behaviors instead of "bad
> substitution" error will
> not hurt zsh even in ksh emulation mode.
>
> Any comments are appreciated.
>
> Thanks,
> Tomoki

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Bart Schaefer]

On Jan 26,  3:00pm, Tomoki Sekiyama wrote:
}
}                     # ksh       bash     zsh(default)       zsh(KSH)
} echo $aaa           # bbb       bbb      bbb                bbb
} echo ${!aaa}        # aaa       ccc      bad substitution   bbb

OK, but the point is that ideally in that last row the zsh(KSH) case
would be "aaa".  I guess this is no worse than the difference of
${array[1]} with or without ksharrays set.  (If "nameref aaa=bbb",
then the first column of the last row would be "bbb".)

}                     # ksh       bash     zsh(default)       zsh(KSH)
} echo ${!foo[bar]}   # foo[bar]  piyo     bad substitution   bar

I find this one interesting, because the ksh doc says it "expands to the
name of the subscript" which would seem to me to fit the zsh(KSH) column
more than the ksh column.

Ksh also has the special case ${!foo[bar..baz]} which treats the array
keys as an alphabetical list and generates a sub-list of them from "bar"
through "baz" (uninteresting in this example).

} According to this, introducing bash-like behaviors instead of "bad
} substitution" error will not hurt zsh even in ksh emulation mode.

My question is, why choose the bash behavior for the zsh default and
not the ksh behavior?  If we make the bash behavior the default, then
also adding namerefs as a default leaves us deviating from ksh on the
meaning of ${!aaa} for identifiers that are not namerefs.

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Tomoki Sekiyama]

2015-01-27 0:48 GMT-05:00 Bart Schaefer <schaefer@brasslantern.com>:
> On Jan 26,  3:00pm, Tomoki Sekiyama wrote:
> }
> }                     # ksh       bash     zsh(default)       zsh(KSH)
> } echo $aaa           # bbb       bbb      bbb                bbb
> } echo ${!aaa}        # aaa       ccc      bad substitution   bbb
>
> OK, but the point is that ideally in that last row the zsh(KSH) case
> would be "aaa".  I guess this is no worse than the difference of
> ${array[1]} with or without ksharrays set.  (If "nameref aaa=bbb",
> then the first column of the last row would be "bbb".)

I agree. So until nameref is introduced, ${!aaa} should always be "aaa" in
ksh emulation mode.

> My question is, why choose the bash behavior for the zsh default and
> not the ksh behavior?  If we make the bash behavior the default, then
> also adding namerefs as a default leaves us deviating from ksh on the
> meaning of ${!aaa} for identifiers that are not namerefs.

I'm not familiar with zsh policy to determine default behaviors, but if
there is future plan (or possibility) to introduce nameref, we certainly
should provide this bash-like indication under an option.
For example:

  $ unsetopt banghist
  $ typeset -A foo
  $ foo[bar]=qux
  $ foo[baz]=piyo
  $ baz=BAZ
  $ echo ${foo[@]}
  qux piyo
  $ echo ${!foo[@]}
  bar baz
  $ setopt bashindirection
  $ echo ${!foo[bar]}
  BAZ
  $ emulate -R ksh
  $ echo ${!foo[bar]}
  foo[bar]

Without these options, we might only support '!' for ${!foo[@]}
getting arrays of the keys of foo.
How do you think of this?

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Bart Schaefer]

On Jan 27, 10:21pm, Tomoki Sekiyama wrote:
}
} I'm not familiar with zsh policy to determine default behaviors, but if
} there is future plan (or possibility) to introduce nameref, we certainly
} should provide this bash-like indication under an option.

There really isn't a policy, it's just sort of by consensus, which is why
I bring it up for discussion.

Re: [PATCH] Enable further expansion of parameter name by ${!...}

[Bart Schaefer]

Noticed patches related to this in my git stash.  The thread dies out
without an answer to the question of which default behavior to choose.

I'll quote the entire year-old last substantive message for reference.

On Jan 26, 2015,  9:48pm, Bart Schaefer wrote:
} Subject: Re: [PATCH] Enable further expansion of parameter name by ${!...}
}
} On Jan 26,  3:00pm, Tomoki Sekiyama wrote:
} }
} }                     # ksh       bash     zsh(default)       zsh(KSH)
} } echo $aaa           # bbb       bbb      bbb                bbb
} } echo ${!aaa}        # aaa       ccc      bad substitution   bbb
} 
} OK, but the point is that ideally in that last row the zsh(KSH) case
} would be "aaa".  I guess this is no worse than the difference of
} ${array[1]} with or without ksharrays set.  (If "nameref aaa=bbb",
} then the first column of the last row would be "bbb".)
} 
} }                     # ksh       bash     zsh(default)       zsh(KSH)
} } echo ${!foo[bar]}   # foo[bar]  piyo     bad substitution   bar
} 
} I find this one interesting, because the ksh doc says it "expands to the
} name of the subscript" which would seem to me to fit the zsh(KSH) column
} more than the ksh column.
} 
} Ksh also has the special case ${!foo[bar..baz]} which treats the array
} keys as an alphabetical list and generates a sub-list of them from "bar"
} through "baz" (uninteresting in this example).
} 
} } According to this, introducing bash-like behaviors instead of "bad
} } substitution" error will not hurt zsh even in ksh emulation mode.
} 
} My question is, why choose the bash behavior for the zsh default and
} not the ksh behavior?  If we make the bash behavior the default, then
} also adding namerefs as a default leaves us deviating from ksh on the
} meaning of ${!aaa} for identifiers that are not namerefs.


The patch I have (workers/34390) implements the bash behavior as the
default, without changing zsh's incorrect ksh emulation.

Obviously we're nowhere close to adding namerefs, so maybe we should just
punt on this until that happens.