Two bug reports: alias/function disagreements; incorrect redirection

Two bug reports: alias/function disagreements; incorrect redirection

[David Hughes]

These appear to be long-standing problems (since v2.6 at least), and 
they're still around as of v4.0.4, and they're not in the known-bugs 
FAQ, and they seem to be platform-independent.

Bug I
=====

Bad Things happen if I assign a name to an alias and then to a function.

     $ which echo
     echo: shell built-in command
     $ alias foo='echo bar'
     $ foo () { echo baz }
     $ which echo
     echo () {
             echo baz
     }

Of course Very Bad Things now happen if I try to use `echo' or `foo'.


Bug II
======

     $ /bin/sh -c 'echo out; echo err >&2' 2>&1 >/dev/null
     err
     $ /bin/sh -c 'echo out; echo err >&2' 2>&1 >/dev/null | cat
     err
     out


I have reproduced both of these bugs on these platforms:

     RedHat 7.2 GNU/Linux (heavily CERN-customised), zsh 4.0.4
     RedHat 7.1 GNU/Linux, zsh 3.1.6-dev-22
     SunOS, zsh 2.6-beta4
     HP-UX, zsh 2.6-beta21

// David
-- 
David Hughes
UNIX sysadmin, Serco SA         -+-         Tel.: +41 22 767 4047
Computing Centre, CERN          -+-        David.W.Hughes@cern.ch

                    +++ A good bug report is an expression of love.

This message expresses my own opinions and should not be construed
as the opinions of Serco (who employ me) or of CERN (where I work).

Re: Two bug reports: alias/function disagreements; incorrect redirection

[Peter Stephenson]

David Hughes wrote:
> Bug I
> =====
> 
> Bad Things happen if I assign a name to an alias and then to a function.
> 
>      $ which echo
>      echo: shell built-in command
>      $ alias foo='echo bar'
>      $ foo () { echo baz }
>      $ which echo
>      echo () {
>              echo baz
>      }
> 
> Of course Very Bad Things now happen if I try to use `echo' or `foo'.

This is inconvenient but is already mentioned in the FAQ
(http://zsh.sunsite.dk/FAQ/), section 2.3.

  There is one other serious problem with aliases: consider 

      alias l='/bin/ls -F'
      l() { /bin/ls -la "$@" | more }

  l in the function definition is in command position and is expanded as
  an alias, defining /bin/ls and -F as functions which call /bin/ls, which
  gets a bit recursive. This can be avoided if you use function to define
  a function, which doesn't expand aliases. It is possible to argue for
  extra warnings somewhere in this mess. Luckily, it is not possible to
  define function as an alias.

(The last sentence is a lie, unfortunately, and I must get around to
changing it.)

> Bug II
> ======
> 
>      $ /bin/sh -c 'echo out; echo err >&2' 2>&1 >/dev/null
>      err
>      $ /bin/sh -c 'echo out; echo err >&2' 2>&1 >/dev/null | cat
>      err
>      out

This is not a bug, and if you are in sh-compatibility mode it doesn't
happen.  From the FAQ again,

  3.26: Why is my output duplicated with `foo 2>&1 >foo.out | bar'?

  This is a slightly unexpected effect of the option MULTIOS, which is set
  by default. Let's look more closely:

      foo 2>&1 >foo.out | bar

  What you're probably expecting is that the command foo sends its
  standard output to the pipe and so to the input of the command bar,
  while it sends its standard error to the file foo.out. What you actually
  see is that the output is going both to the pipe and into the file. To
  be more explicit, here's the same example with real commands:

      % { print output; print error >&2 } 2>&1 >foo.out | sed 's/error/erratic'
      erratic
      output
      % cat foo.out
      output

  and you can see `output' appears twice.

  It becomes clearer what's going on if we write: 

      % print output >foo1.out >foo2.out
      % cat foo1.out
      output
      % cat foo2.out
      output

  You might recognise this as a standard feature of zsh, called `multios'
  and controlled by the option of the same name, whereby output is copied
  to both files when the redirector appears twice. What's going on in the
  first example is exactly the same, however the second redirector is
  disguised as a pipe. So if you want to turn this effect off, you need to
  unset the option MULTIOS.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR Ltd., Science Park, Milton Road,
Cambridge, CB4 0WH, UK                          Tel: +44 (0)1223 392070


**********************************************************************
The information transmitted is intended only for the person or
entity to which it is addressed and may contain confidential 
and/or privileged material. 
Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by 
persons or entities other than the intended recipient is 
prohibited.  
If you received this in error, please contact the sender and 
delete the material from any computer.
**********************************************************************

Piping stderr (was Re: Two bug reports)

[Wayne Davison]

I'm wondering if it would be nice to create a syntax for piping just
stderr?  We already have "|&" for piping both stdout and stderr, so
perhaps we could extend this somehow?  What do you think about using
"2|&"?  Or maybe "2>|&" (which is probably safer).  Do any other shells
have a syntax for this?

..wayne..

Re: Piping stderr (was Re: Two bug reports)

[Bart Schaefer]

On Mar 23,  1:18pm, Wayne Davison wrote:
}
} I'm wondering if it would be nice to create a syntax for piping just
} stderr?

You mean, other than `2>>(...)'?

} We already have "|&" for piping both stdout and stderr

Hmm.  `|&' is short for `2>&1 |'.  What would the new syntax be short
for?  The problem being that in order to pipe "just stderr" you have
to dispose of stdout somehow.

The proposed new syntax could be short for `2>&1 >&- |' but then you
may get "bad file descriptor" because of the closed stdout.

Or it could be short for `2>&1 >/dev/null |' except that in the case
of MULTIOS that doesn't redirect stdout, it duplicates it to both
/dev/null and the pipe; we'd at the least have to document as we do
for `>&' that the effect of MULTIOS is subverted.  Plus, some platforms
don't have a /dev/null (zsh does compile for DOS, sort of).

The same problem with MULTIOS occurs with `3>&2 2>&1 1>&3 |'.

} What do you think about using "2|&"?  Or maybe "2>|&"

Of those, it'd have to be the latter, which is currently a parse error.
The issue is really one of syntactic cleanliness; `|', `&', and `|&'
are all command separators, whereas redirections such as `2>&1' are
part of the command itself.  That's why `2>&1 |' works differently
than `2>&1 >'.  The grammar gets a bit ugly if you make something that
starts with a digit into a command separator.

} Do any other shells have a syntax for this?

Not that I know of.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Multios again (Re: Piping stderr (was Re: Two bug reports))

[Bart Schaefer]

On Mar 23, 10:45pm, Bart Schaefer wrote:
}
} The same problem with MULTIOS occurs with `3>&2 2>&1 1>&3 |'.

Given that we have the >(...) syntax, I'm wondering there should be a way
to allow multios for redirections but not for pipes.  That would mean that
in `command >file1 >file2 | prog', file1 and file2 would get output but the
pipe would get nothing.  That way, `command 2>&1 >file | prog' would work
the way it does in other shells, and if you really mean for `prog' to get
a copy of the input you can write `command 2>&1 >file >>(prog)' instead.

It further would mean that `command 2>>(prog1) >file | prog2' would send
ONLY the output of `prog1' through `prog2', which seems to be partly what
Wayne was getting at with his question.

Various weird tidbits while I'm on the subject:

Using `1>&1' creates some pretty strange effects; it duplicates stdout
2**N times (where N is the number of uses of `1>&1').  Let's look at a few
examples; the first two just set up the context.

zsh% (print -u2 foo; print bar) >>(tr a-z A-Z)
foo
zsh% BAR

We know that the >(...) runs asynchronously, so that's fine.

zsh% (print -u2 foo; print bar) >>(tr a-z A-Z) | wc
foo
      2       2       8

Shows that the stdout of the >(...) expression also goes to the pipe.  That
implies a simple way to make >(...) act semi-synchronously: pipe the whole
command to `cat'.

Now for the strange stuff:

zsh% (print -u2 foo; print bar) 1>&1 >>(tr a-z A-Z) | cat
foo
bar
bar
BAR

So `cat' got two copies of the original stdout and >(...) got one.  Yet:

zsh% (print -u2 foo; print bar) >>(tr a-z A-Z) 1>&1 | cat
foo
bar
bar
BAR
BAR

Here, 1>&1 duplicated the input to >(...) as well as to the pipe.  At first
I thought it had somehow duplicated the output of >(...), but:

zsh% (print -u2 foo; print bar) 2>>(tr a-z A-Z) 1>&1 | cat
bar
bar
FOO

Now for the real silliness:

zsh% (print -u2 foo; print bar) 1>&1 >>(tr a-z A-Z) 1>&1 | cat
foo
bar
bar
bar
bar
BAR
BAR

Can this really be the way it's supposed to work?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Piping stderr (was Re: Two bug reports)

[Wayne Davison]

On Sat, 23 Mar 2002, Bart Schaefer wrote:
> On Mar 23,  1:18pm, Wayne Davison wrote:
> } I'm wondering if it would be nice to create a syntax for piping just
> } stderr?
>
> You mean, other than `2>>(...)'?

Uh ...  Yeah, Yeah.  That's what I meant.  _Other_ than that.  :-)

(I had apparently neglected to find that syntax when I was searching
the zsh man page.)

So, I suppose that Peter's reply to David Hughes could be ammended to
say that in addition to turning off multios, he could change this
command:

    /bin/sh -c 'echo out; echo err >&2' 2>&1 >/dev/null | cat

into this one:

    /bin/sh -c 'echo out; echo err >&2' >/dev/null 2>>(cat)

The bad thing about this idiom is that it doesn't wait for "cat" to
finish before moving on.  If we implemented my proposed stderr-pipe
syntax, the following command wouldn't have that problem:

    /bin/sh -c 'echo out; echo err >&2' >/dev/null 2>|& cat

> Hmm.  `|&' is short for `2>&1 |'.  What would the new syntax be short
> for?

It wouldn't be short for anything.  Instead "|" would become short for
"1>|&".  The construct simply means "take the numbered file handle and
pipe it to stdin of the following command."

> The problem being that in order to pipe "just stderr" you have
> to dispose of stdout somehow.

No, stdout doesn't need to be affected at all.  If the user wants stdout
to go somewhere other than the default, it can be redirected explicitly
(as I did above).

..wayne..

Re: Piping stderr (was Re: Two bug reports)

[Borsenkow Andrej]

В Вск, 24.03.2002, в 12:01, Wayne Davison написал:
> 
> > The problem being that in order to pipe "just stderr" you have
> > to dispose of stdout somehow.
> 
> No, stdout doesn't need to be affected at all.  If the user wants stdout
> to go somewhere other than the default, it can be redirected explicitly
> (as I did above).
> 

That makes sense. More than once I wished to pipe stderr while leaving
stdout alone.

-andrej

Re: Piping stderr (was Re: Two bug reports)

[Bart Schaefer]

On Mar 24,  1:01am, Wayne Davison wrote:
} Subject: Re: Piping stderr (was Re: Two bug reports)
}
} On Sat, 23 Mar 2002, Bart Schaefer wrote:
} >
} > You mean, other than `2>>(...)'?
} 
} (I had apparently neglected to find that syntax when I was searching
} the zsh man page.)

The man page could stand improvement in this area.  Even though pipes are
not redirections, people tend to think of them that way, and it's very
confusing that >(...) and |& are not at least cross-referenced in the
same section as all the redirection operators.  Even `|&' is buried in
the middle of a boring-looking paragraph in the Shell Grammar section,
and never metioned again.
 
} So, I suppose that Peter's reply to David Hughes could be ammended to
} 
}     /bin/sh -c 'echo out; echo err >&2' >/dev/null 2>>(cat)
} 
} The bad thing about this idiom is that it doesn't wait for "cat" to
} finish before moving on.

That's a separate bug that I've been complaining about for a while.  I
really think zsh should be tracking >(...) as a job and waiting for it
rather than treating it as if disowned.  That would be a much cleaner
solution from the viewpoint of the shell language (though possibly a bit
messier internally) than would adding a new pipe syntax.

As I pointed out in a follow-up you can work around it by doing an extra
pipe:

    /bin/sh -c 'echo out; echo err >&2' >/dev/null 2>>(cat) | cat

In fact (hand smacks forehead), just add braces:

    { /bin/sh -c 'echo out; echo err >&2' >/dev/null 2>>(cat) }

} > The problem being that in order to pipe "just stderr" you have
} > to dispose of stdout somehow.
} 
} No, stdout doesn't need to be affected at all.  If the user wants stdout
} to go somewhere other than the default, it can be redirected explicitly
} (as I did above).

Leaving stdout connected to the parent's stdout is still "disposing of"
stdout; it's equivalent to:

zsh% 3>&1 { { /bin/sh -c 'echo out; echo err >&2' >&3 } |& tr a-z A-Z }
out
ERR

(The placement of `3>&1' there is illustrative, it could just as easily
go after the last close brace.)  You can always defeat multios by adding
braces.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Piping stderr (was Re: Two bug reports)

[Bart Schaefer]

On Mar 24,  7:02pm, Bart Schaefer wrote:
}
} In fact (hand smacks forehead), just add braces:
} 
}     { /bin/sh -c 'echo out; echo err >&2' >/dev/null 2>>(cat) }

Oops, sorry, that doesn't work unless you also redirect the whole
brace expression somewhere.  I confused myself.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

PATCH: (Doc) Re: Piping stderr (was Re: Two bug reports)

[Bart Schaefer]

On Mar 24,  7:02pm, Bart Schaefer wrote:
}
} The man page could stand improvement in this area.  Even though pipes are
} not redirections, people tend to think of them that way, and it's very
} confusing that >(...) and |& are not at least cross-referenced in the
} same section as all the redirection operators.  Even `|&' is buried in
} the middle of a boring-looking paragraph in the Shell Grammar section,
} and never metioned again.

Anybody object to this?

Index: Doc/Zsh/redirect.yo
===================================================================
diff -c -r1.2 redirect.yo
--- Doc/Zsh/redirect.yo	2001/07/10 09:05:18	1.2
+++ Doc/Zsh/redirect.yo	2002/03/24 19:45:25
@@ -138,6 +138,19 @@
 file descriptor 2 would be associated
 with the terminal (assuming file descriptor 1 had been)
 and then file descriptor 1 would be associated with file var(fname).
+
+The `tt(|&)' command separator described in
+ifzman(em(Simple Commands & Pipelines) in zmanref(zshmisc))\
+ifnzman(noderef(Simple Commands & Pipelines))
+is a shorthand for `tt(2>&1 |)'.
+
+For output redirections only, if var(word) is of the form
+`tt(>LPAR())var(list)tt(RPAR())' then the output is piped to the command
+represented by var(list).  See
+ifzman(\
+em(Process Substitution) in zmanref(zshexpn))\
+ifnzman(\
+noderef(Process Substitution)).
 sect(Multios)
 cindex(multios)
 pindex(MULTIOS, use of)


-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Piping stderr (was Re: Two bug reports)

[Wayne Davison]

On Sun, 24 Mar 2002, Bart Schaefer wrote:
> I really think zsh should be tracking >(...) as a job and waiting for
> it rather than treating it as if disowned.

Yes, that would be a good change, IMO.

> That would be a much cleaner solution from the viewpoint of the shell
> language (though possibly a bit messier internally) than would adding
> a new pipe syntax.

I agree.  After looking at my proposed stderr-pipe syntax, I don't like
it very much.  So, using the existing 2>>(command) syntax is a good way
to go, especially if zsh could be made to wait for the "command" to
complete.

..wayne..