ZSH expansion of an evaluated parameter containing a pipe character (and differences between 4.x and 5.x)

ZSH expansion of an evaluated parameter containing a pipe character (and differences between 4.x and 5.x)

[Enrico Maria Crisostomo]

Hi all,

I’m trying to understand the rationale between a behavioural difference between ZSH 4.3.17 (current in latest Debian stable) and ZSH 5.0.2 (current in latest OS X).  I’m digging into the 'ZSH Shell Manual’ but I haven’t found the (exact) answer to my question yet.

The issue is the following.  What I’m doing is storing into a variable a command that will be executed later, the reason being the command depend on information only available at runtime.  I’ve detected a difference in behaviour between ZSH 4 and ZSH 5 when the aforementioned command contains a pipe.  For the sake of argument, let’s imagine the command is stored into `EXAMPLE_CMD` and is as simple as:

    EXAMPLE_CMD=“ls -al | grep text”

If I eval this variable using ZSH 5, the result is the expected:

    $ eval ${EXAMPLE_CMD}

that is:

  * `ls` is executed and passed `-al` as a parameter.
  * Its output is piped through `grep`, that is passed `text` as a parameter.

When I do the same thing in ZSH 4, the behaviour is different and leads to this failure:

    # eval ${EXAMPLE_CMD}
    zsh: no matches found: ls -al | grep text

However, if I quote `EXAMPLE_CMD` in either version of the shell, the result is the same:

    # eval “${EXAMPLE_CMD}”
    drwxr-xr-x  2 root root  4096 Oct  6 21:02 text

According to my observations, it seems this issue is limited to the interpretation of the pipe in the command, since I’ve found no other differences between the behaviours of the two shell versions in this case.

To me, it came quite as a surprise to discover that quoting worked, since I assumed that eval would receive 1 argument (the whole quoted string) and that its behaviour wouldn’t change.  Which leads me to think that either ZSH 4 is performing an expansion I’m not aware of affecting (at least) the meaning of `|`, or that this behaviour is buggy.

Now, the first question that I’m trying to answer is “Why expansion is done like that?”.  The second one that comes to mind is: what’s the rationale behind this change?  Is the behaviour of ZSH 4 considered buggy?

More generally: could you share your thoughts about the “proper”/“best”/“put your favourite adjective here" way to handle this case in ZSH?

Thank you very much for your insights.

Cheers,
--  
Enrico Maria Crisostomo

Re: ZSH expansion of an evaluated parameter containing a pipe character (and differences between 4.x and 5.x)

[Peter Stephenson]

On Thu, 09 Oct 2014 09:41:14 +0200
Enrico Maria Crisostomo <enrico.m.crisostomo@gmail.com> wrote:
> I’m trying to understand the rationale between a behavioural
> difference between ZSH 4.3.17 (current in latest Debian stable) and
> ZSH 5.0.2 (current in latest OS X).  I’m digging into the 'ZSH Shell
> Manual’ but I haven’t found the (exact) answer to my question yet.
> 
> The issue is the following.  What I’m doing is storing into a variable
> a command that will be executed later, the reason being the command
> depend on information only available at runtime.  I’ve detected a
> difference in behaviour between ZSH 4 and ZSH 5 when the
> aforementioned command contains a pipe.  For the sake of argument,
> let’s imagine the command is stored into `EXAMPLE_CMD` and is as
> simple as:
> 
>     EXAMPLE_CMD=“ls -al | grep text”
> 
> If I eval this variable using ZSH 5, the result is the expected:
> 
>     $ eval ${EXAMPLE_CMD}
> 
> that is:
> 
>   * `ls` is executed and passed `-al` as a parameter.
>   * Its output is piped through `grep`, that is passed `text` as a parameter.
> 
> When I do the same thing in ZSH 4, the behaviour is different and leads to this failure:
> 
>     # eval ${EXAMPLE_CMD}
>     zsh: no matches found: ls -al | grep text

I suspect you've got the GLOB_SUBST option set in the second case but
not the first --- if I set that in either version of the shell
(4.3.10-dev-1 is the oldest I had lying around) I get that error.
If I unset it I don't.

That's because in zsh "|" is a pattern match character as well as a
syntactical character.  In a normal command line you'd surround it with
parentheses to use it for pattern matching.  However, when it's being
substituted into the command line, so the line isn't being parsed, and
GLOB_SUBST is set, it's treated as a pattern character immediately.

GLOB_SUBST is usually used for compatibility with other shells with
different pattern matching behaviour.  If you set the option SH_GLOB as
well, which restricts pattern matching to standard sh-style patterns,
you should find the error goes away --- it does here in both versions of
the shell.

Basically, I think you need to decide if you want native zsh operation
--- both options off --- or sh-like operation --- both options on.
Arguably there are too many combinations, for historical reasons.

pws

Re: ZSH expansion of an evaluated parameter containing a pipe character (and differences between 4.x and 5.x)

[Enrico Maria Crisostomo]

[-- Attachment #1: Type: text/plain, Size: 3776 bytes --]

Hi Stephen,

Doh! I think you nailed it! I didn’t notice GLOB_SUBST was set in the second case: I made all the tests you mentioned and the result is as expected.

I confess there was a moment when GLOB_SUBST came to mind, but dismissed it: I (wrongly) thought it was only used for filename expansion. Yesterday I went through your “A User’s Guide to the Z-Shell” (I always keep it close, it’s an excellent guide, kudos for your great work), and failed to realise that pattern matching could have a part in what was happening.

I already decided what I’ll do: I’ll stick with ZSH’s native operation, which I find way more compatible with my way of thinking (and, why not, even shell script aesthetics), which are the reason why I’m a ZSH user.

If you don’t mind, I’d suggest adding more references to the effect of GLOB_SUBST on pattern matching (maybe with an example). Although the effect is stated clearly (Section 5.4.5 “Flags for options: GLOB_SUBST and RC_EXPAND_PARAM”), as far as I can see all the other references and examples involving GLOB_SUBST only talk about filename expansion.

Thank you very much!
-- 
Enrico Maria Crisostomo

On 9 Oct 2014 at 12:49:22, Peter Stephenson (p.stephenson@samsung.com) wrote:

On Thu, 09 Oct 2014 09:41:14 +0200  
Enrico Maria Crisostomo <enrico.m.crisostomo@gmail.com> wrote:  
> I’m trying to understand the rationale between a behavioural  
> difference between ZSH 4.3.17 (current in latest Debian stable) and  
> ZSH 5.0.2 (current in latest OS X).  I’m digging into the 'ZSH Shell  
> Manual’ but I haven’t found the (exact) answer to my question yet.  
>  
> The issue is the following.  What I’m doing is storing into a variable  
> a command that will be executed later, the reason being the command  
> depend on information only available at runtime.  I’ve detected a  
> difference in behaviour between ZSH 4 and ZSH 5 when the  
> aforementioned command contains a pipe.  For the sake of argument,  
> let’s imagine the command is stored into `EXAMPLE_CMD` and is as  
> simple as:  
>  
>     EXAMPLE_CMD=“ls -al | grep text”  
>  
> If I eval this variable using ZSH 5, the result is the expected:  
>  
>     $ eval ${EXAMPLE_CMD}  
>  
> that is:  
>  
>   * `ls` is executed and passed `-al` as a parameter.  
>   * Its output is piped through `grep`, that is passed `text` as a parameter.  
>  
> When I do the same thing in ZSH 4, the behaviour is different and leads to this failure:  
>  
>     # eval ${EXAMPLE_CMD}  
>     zsh: no matches found: ls -al | grep text  

I suspect you've got the GLOB_SUBST option set in the second case but  
not the first --- if I set that in either version of the shell  
(4.3.10-dev-1 is the oldest I had lying around) I get that error.  
If I unset it I don't.  

That's because in zsh "|" is a pattern match character as well as a  
syntactical character. In a normal command line you'd surround it with  
parentheses to use it for pattern matching. However, when it's being  
substituted into the command line, so the line isn't being parsed, and  
GLOB_SUBST is set, it's treated as a pattern character immediately.  

GLOB_SUBST is usually used for compatibility with other shells with  
different pattern matching behaviour. If you set the option SH_GLOB as  
well, which restricts pattern matching to standard sh-style patterns,  
you should find the error goes away --- it does here in both versions of  
the shell.  

Basically, I think you need to decide if you want native zsh operation  
--- both options off --- or sh-like operation --- both options on.  
Arguably there are too many combinations, for historical reasons.  

pws