Modifiers and parameter expansion?

Modifiers and parameter expansion?

[Dominik Vogt]

Why are modifiers applied here; is this a bug?

  $ zsh --version
  zsh 5.8 (x86_64-debian-linux-gnu)
  $ emulate zsh
  $ X=abc
  $ echo $X:u
  ABC

Isn't this supposed to happen only if the parameter and modifier
are put in braces?

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Modifiers and parameter expansion?

[Eric Cook]

On 12/1/22 16:29, Dominik Vogt wrote:
> Why are modifiers applied here; is this a bug?
they are defined to work that way, no.
> 
>    $ zsh --version
>    zsh 5.8 (x86_64-debian-linux-gnu)
>    $ emulate zsh
>    $ X=abc
>    $ echo $X:u
>    ABC
> 
> Isn't this supposed to happen only if the parameter and modifier
> are put in braces?
> 
nope.

Re: Modifiers and parameter expansion?

[Bart Schaefer]

On Thu, Dec 1, 2022 at 1:34 PM Eric Cook <llua@gmx.com> wrote:
>
> On 12/1/22 16:29, Dominik Vogt wrote:
> > Isn't this supposed to happen only if the parameter and modifier
> > are put in braces?
> >
> nope.

More specifically, Dominik is thinking in bash, whereas zsh is copying
implementation from csh.

Re: Modifiers and parameter expansion?

[Dominik Vogt]

On Thu, Dec 01, 2022 at 01:36:53PM -0800, Bart Schaefer wrote:
> On Thu, Dec 1, 2022 at 1:34 PM Eric Cook <llua@gmx.com> wrote:
> >
> > On 12/1/22 16:29, Dominik Vogt wrote:
> > > Isn't this supposed to happen only if the parameter and modifier
> > > are put in braces?
>
> More specifically, Dominik is thinking in bash, whereas zsh is copying
> implementation from csh.

Actually I've never used anything but zsh unless forced to and
never tried to understand any other shell.

I cannot find anything in the man page that suggest it would work
this way.  All occurences of modifiers used with parameters put
them in braces.  The rule from the man page:

       7. Modifiers
              Any modifiers, as specified by a trailing `#', `%', `/'  (possi-
              bly  doubled)  or  by a set of modifiers of the form `:...' (see
              the section `Modifiers' in the section `History Expansion'), are
              applied to the words of the value at this level.

If this rule applies to "$X:...", what is "$X%..." supposed to do?

  $ X=abc
  $ echo "$X:u" "$X%c" "${X%c}"
  ABC abc%c ab

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Modifiers and parameter expansion?

[Bart Schaefer]

On Thu, Dec 1, 2022 at 2:39 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> I cannot find anything in the man page that suggest it would work
> this way.  All occurences of modifiers used with parameters put
> them in braces.  The rule from the man page:
>
>        7. Modifiers

That rule is just telling you at what point during all possible
expansions the effects of modifiers come into play, it's unrelated to
the syntax.

In the Parameter Expansion section

${NAME}
     The value, if any, of the parameter NAME is substituted.  The
     braces are required if the expansion is to be followed by a letter,
     digit, or underscore that is not to be interpreted as part of NAME.
     In addition, more complicated forms of substitution usually require
     the braces to be present; exceptions, which only apply if the
     option KSH_ARRAYS is not set, are a single subscript or any colon
     modifiers appearing after the name, or any of the characters '^',
     '=', '~', '#' or '+' appearing before the name, all of which work
     with or without braces.

Note "exceptions ... are a single subscript or any colon modifiers
appearing after the name ..."

Re: Modifiers and parameter expansion?

[Dominik Vogt]

On Thu, Dec 01, 2022 at 03:42:27PM -0800, Bart Schaefer wrote:
> On Thu, Dec 1, 2022 at 2:39 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> >
> > I cannot find anything in the man page that suggest it would work
> > this way.  All occurences of modifiers used with parameters put
> > them in braces.  The rule from the man page:
> >
> >        7. Modifiers
>
> That rule is just telling you at what point during all possible
> expansions the effects of modifiers come into play, it's unrelated to
> the syntax.

Okay.  Why are "%" "%%" "#" "##" "/" "//" refered to in this
section?  They're not called "modifiers" anywhere else.  Maybe the
rule should be named "Pattern replacements and modifiers"?

> In the Parameter Expansion section
>
> ${NAME}
>      The value, if any, of the parameter NAME is substituted.  The
>      braces are required if the expansion is to be followed by a letter,
>      digit, or underscore that is not to be interpreted as part of NAME.
>      In addition, more complicated forms of substitution usually require
>      the braces to be present; exceptions, which only apply if the
>      option KSH_ARRAYS is not set, are a single subscript or any colon
>      modifiers appearing after the name, or any of the characters '^',
>      '=', '~', '#' or '+' appearing before the name, all of which work
>      with or without braces.
>
> Note "exceptions ... are a single subscript or any colon modifiers
> appearing after the name ..."

The braces are required unless(1) they are not(2) required
unless(3) some option is not(4) present.  Quadruple negative.

All I understand is that it's trial and error to figure out
whether braces are required.  Let's see:

  $ rm -rf tmp
  $ mkdir tmp
  $ cd tmp
  $ touch a b

  $ unsetopt KSH_ARRAYS
  $ echo ^a =ls
  b /bin/ls                                <-- matches man page
  $ echo ~a
  zsh: no such user or named directory: a  <-- matches man page
  $ X=abc; echo $X:u $X[2]
  ABC B                                    <-- matches man page
  $ echo $+X $#X
  1 3                                      <-- matches man page

  $ setopt KSH_ARRAYS
  $ echo ^a =ls
  b /bin/ls                      <-- not what the man page says
  $ echo ~a                                     |
  zsh: no such user or named directory: a  <----|
  $ X=abc; echo $X:u                            |
  ABC                            <-- ok         |
  $ echo $X[2]                                  |
  zsh: no matches found: ABC[2]  <-- ok         |
  $ echo $+X $#X                                |
  1 3                            <--------------

--

This section could be split in two, to improve readability:

  ${NAME}
       The value, if any, of the parameter NAME is substituted.
       This form is required if the characters following the closing
       brace would be interpreted as part of the substitution
       otherwise.

  $NAME
       The value, if any, of the parameter NAME is substituted.  A
       letter, digit or underscore after NAME is interpreted as part
       of NAME. The characters '^', '=', '~', '#' or '+' appearing
       before NAME work without braces.  A single subscript or any
       colon modifiers appearing after NAME work only if the
       KSH_ARRAYS option is not set.  All other kinds of complex
       substitutions require the form with braces.  Using braces is
       always safe.

And adding a short section about modifiers may be a good idea:

  PARAMETER EXPANSION
      ...

    Modifiers
      Modifiers preceded ba a `:' as described in  the  section
      `Modifiers' in the section `History Expansion' can be applied.
      If the KSH_ARRAYS option is set, braces are required to use
      modifiers.

    Parameter Expansion Flags
      ...

It's a slight duplication of information, but much easier to find
for normal mortals.  (Note that the `:' is there on purpose to
make it easier to find when searching through the man page.)

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Modifiers and parameter expansion?

[Bart Schaefer]

On Thu, Dec 1, 2022 at 4:52 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> > >        7. Modifiers
>
> Okay.  Why are "%" "%%" "#" "##" "/" "//" refered to in this
> section?

Because they happen to be interpreted at the same "step" as modifiers,
and the documentation grew organically?

> > In the Parameter Expansion section
> >
> > ${NAME}
>
> The braces are required unless(1) they are not(2) required
> unless(3) some option is not(4) present.  Quadruple negative.

Not sure where you came up with that.  The braces are required when
(A) or when (B) or when (C), except (D) when not (C).
(A) followed by something may be part of an identifier
(B) complicated forms
(C) KSH_ARRAYS
(D) colons after or other tokens before

> All I understand is that it's trial and error to figure out
> whether braces are required.

Nonsense.

>   $ unsetopt KSH_ARRAYS
>   $ echo ^a =ls
>   $ echo ~a

No, that's completely irrelevant.  The BRACES aren't required, but the
$ IS.  The above is not parameter expansion.

NO_KSH_ARRAYS allows
$X[2]
$X:h
$^X
$=X
$~X
$#X
$+X

KSH_ARRAYS requires
${X[2]}
${X:h}
${^X}
${=X}
${~X}
${#X}
${+X}

> This section could be split in two, to improve readability:

Possibly, but that affects several of the following descriptions as well.

Re: Modifiers and parameter expansion?

[Dominik Vogt]

On Thu, Dec 01, 2022 at 05:34:18PM -0800, Bart Schaefer wrote:
> On Thu, Dec 1, 2022 at 4:52 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> >
> > > >        7. Modifiers
> >
> > Okay.  Why are "%" "%%" "#" "##" "/" "//" refered to in this
> > section?
>
> Because they happen to be interpreted at the same "step" as modifiers,
> and the documentation grew organically?
>
> > > In the Parameter Expansion section
> > >
> > > ${NAME}
> >
> > The braces are required unless(1) they are not(2) required
> > unless(3) some option is not(4) present.  Quadruple negative.
>
> Not sure where you came up with that.

  [Implicit: the braces can be omitted, unless]         -> negation 1
  ... more complicated forms ... require ... exceptions -> negation 2
  ... which only apply if                               -> negation 3
  ... option is not set                                 -> negation 4

(The implicit statement is the _only_ "mention" of the brace-less
form at all.)

--

> >   $ echo ~a
>
> No, that's completely irrelevant.  The BRACES aren't required, but the
> $ IS.  The above is not parameter expansion.

Right, sorry.

--

> KSH_ARRAYS requires
> ${X[2]}
> ${X:h}
> ${^X}

True.

> ${=X}
> ${~X}
> ${#X}
> ${+X}

These work without braces even if KSH_ARRAYS is set.

  $ emulate zsh; setopt KSH_ARRAYS

  ### $=X
  $ X='abc def'
  $ A=("$X"); echo ${#A[@]}
  1
  $ B=("$=X"); echo ${#B[@]}
  2
  $ unsetopt KSH_ARRAYS
  $ C=("$=X"); echo ${#C[@]}
  2
  $ setopt KSH_ARRAYS

  ### $~X
  $ mkdir tmp; cd tmp; touch a b
  $ A='*'; echo $~A; echo $A
  a b
  *

  ### $+X
  $ A="val"; unset B; echo $+A $+B
  1 0

Still works (but ksh-array-ish for arrays):

  ### $#X
  $ A=(aa bb cc); B="abc"; echo $#A $#B
  2 3

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Modifiers and parameter expansion?

[Bart Schaefer]

On Thu, Dec 1, 2022 at 6:48 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> On Thu, Dec 01, 2022 at 05:34:18PM -0800, Bart Schaefer wrote:
> > Not sure where you came up with that.
>
>   [Implicit: the braces can be omitted, unless]

OK ... but that didn't come from the context in the doc, which uses
the form with braces ${NAME} as the base case.

> (The implicit statement is the _only_ "mention" of the brace-less
> form at all.)

"with or without braces" is explicitly in the last sentence?  But
again the manual is largely written from the viewpoint of someone who
knows how shells work and therefore is familiar with the $NAME case
and needs to be told the way zsh might diverge from that.

> > ${=X}
> > ${~X}
> > ${#X}
> > ${+X}
>
> These work without braces even if KSH_ARRAYS is set.

Hmm, I would say that's a bug, then, in "emulate sh" if not in
KSH_ARRAYS.  $#X should be interpreted as ${#}X (and is, with "emulate
sh", even though there's no specific setopt controlling it).  Bash
doesn't even attempt to expand the other three, they're treated as not
being substitutions at all; you get an error only on e.g. ${+X}.

I suppose allowing them to work without braces is an extension in the
same way that supporting them with braces is an extension.

Re: Modifiers and parameter expansion?

[Dominik Vogt]

More inaccuracies in the man page:

On Thu, Dec 01, 2022 at 03:42:27PM -0800, Bart Schaefer wrote:
> ${NAME}
>      The value, if any, of the parameter NAME is substituted.  The
>      braces are required if the expansion is to be followed by a letter,
>      digit, or underscore that is not to be interpreted as part of NAME.

Actually that is only true for "normal" parameters.  Neither of the
special parameters except "_" treats the following characters as
part of the name.

  $ A=abc
  $ echo $?A $@A $*A $-A $$A $!A $#A
  0A A A 0569EKNXZghikmsA 5775A 0A 3

>      In addition, more complicated forms of substitution usually require
>      the braces to be present; exceptions, which only apply if the
>      option KSH_ARRAYS is not set, are a single subscript or any colon
>      modifiers appearing after the name

This is fishy.  With braces, unknown modifiers are rejected,
without braces they are used as literal text:

  $ unsetopt KSH_ARRAYS
  $ X=abc; echo $X:Z
  abc:Z
  $ echo ${X:Z}
  zsh: unrecognized modifier `Z'

Should at least be documented.

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Modifiers and parameter expansion?

[Bart Schaefer]

On Thu, Dec 1, 2022 at 5:28 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> > ${NAME}
> >      The value, if any, of the parameter NAME is substituted.  The
> >      braces are required if the expansion is to be followed by a letter,
> >      digit, or underscore that is not to be interpreted as part of NAME.
>
> Actually that is only true for "normal" parameters.  Neither of the
> special parameters except "_" treats the following characters as
> part of the name.

The problem here is that the man page is written more colloquially
than the interpretation you're trying to put on it.  To do this
rigorously, we'd reference the grammar of an "identifier" and explain
that NAME is the maximal string of characters matching that grammar.
But that would require even more back-referencing and technical jargon
than other people already complain about.

Special parameters aren't identifiers at all, so they don't obey this
description.

> With braces, unknown modifiers are rejected,
> without braces they are used as literal text:

You've caught another place where the manual was written 30 years ago
for people who already knew how csh works.  On the list it goes for
someone with a lot of time, sigh.