Redirecting shell output to a pipe

Redirecting shell output to a pipe

[Nadav Har'El]

Hi,

If a shell script wants all its output to go to a file, it's easy:

	exec >filename 2>&1

Isn't it natural to assume that in the same fashion, you should also
be able to redirect the scripts output to a pipe? E.g., a very useful
idiom could have been

	exec | tee filename

to redirect stdout both to a file, and to the terminal.
But unfortunately, the above line does *not* work. Is there a reason why
it doesn't? Wouldn't this be a useful feature?

By the way, for the curious, there is actually a different solution to this
need, but it is much more convoluted. The other solution is to write this:

	exec 3>&1
	coproc tee /tmp/b >&3
	exec >&p 2>&1

(and perhaps do something, I'm not even sure what, at the end of the script
to give the tee coprocess enough time to finish outputting before the script
exits).

Another downside of this solution, besides being extremely complex (99%
of the zsh users will probably not be able to figure it out), is that
it takes the only coprocess that zsh gives you, which won't work if your
code is already using coprocesses.

-- 
Nadav Har'El                        |      Tuesday, Nov 24 2009, 7 Kislev 5770
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |"Guests, like fish, begin to smell after
http://nadav.harel.org.il           |three days." -- Benjamin Franklin

Re: Redirecting shell output to a pipe

[Peter Miller]

Nadav Har'El wrote:
> Hi,
>
> If a shell script wants all its output to go to a file, it's easy:
>
> 	exec >filename 2>&1
>
> Isn't it natural to assume that in the same fashion, you should also
> be able to redirect the scripts output to a pipe? E.g., a very useful
> idiom could have been
>
> 	exec | tee filename
>   

exec |& tee filename

> to redirect stdout both to a file, and to the terminal.
> But unfortunately, the above line does *not* work. Is there a reason why
> it doesn't? Wouldn't this be a useful feature?
>
> By the way, for the curious, there is actually a different solution to this
> need, but it is much more convoluted. The other solution is to write this:
>
> 	exec 3>&1
> 	coproc tee /tmp/b >&3
> 	exec >&p 2>&1
>
> (and perhaps do something, I'm not even sure what, at the end of the script
> to give the tee coprocess enough time to finish outputting before the script
> exits).
>
> Another downside of this solution, besides being extremely complex (99%
> of the zsh users will probably not be able to figure it out), is that
> it takes the only coprocess that zsh gives you, which won't work if your
> code is already using coprocesses.
>
>   

Re: Redirecting shell output to a pipe

[Joke de Buhr]

[-- Attachment #1: Type: Text/Plain, Size: 1400 bytes --]

On Tuesday 24 November 2009 19:33:45 Peter Miller wrote:
> Nadav Har'El wrote:
> > Hi,
> >
> > If a shell script wants all its output to go to a file, it's easy:
> >
> > 	exec >filename 2>&1
> >
> > Isn't it natural to assume that in the same fashion, you should also
> > be able to redirect the scripts output to a pipe? E.g., a very useful
> > idiom could have been
> >
> > 	exec | tee filename
> 
> exec |& tee filename

bourne shell compatible syntax:

  exec 2>&1 | tee filename

> 
> > to redirect stdout both to a file, and to the terminal.
> > But unfortunately, the above line does *not* work. Is there a reason
> > why it doesn't? Wouldn't this be a useful feature?
> >
> > By the way, for the curious, there is actually a different solution
> > to this need, but it is much more convoluted. The other solution is
> > to write this:
> >
> > 	exec 3>&1
> > 	coproc tee /tmp/b >&3
> > 	exec >&p 2>&1
> >
> > (and perhaps do something, I'm not even sure what, at the end of the
> > script to give the tee coprocess enough time to finish outputting
> > before the script exits).
> >
> > Another downside of this solution, besides being extremely complex
> > (99% of the zsh users will probably not be able to figure it out), is
> > that it takes the only coprocess that zsh gives you, which won't work
> > if your code is already using coprocesses.
> 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Redirecting shell output to a pipe

[Nadav Har'El]

On Tue, Nov 24, 2009, Peter Miller wrote about "Re: Redirecting shell output to a pipe":
>...
> > Isn't it natural to assume that in the same fashion, you should also
> > be able to redirect the scripts output to a pipe? E.g., a very useful
> > idiom could have been
> >
> > 	exec | tee filename
> 
> exec |& tee filename

Does this work for you? I get (on zsh 4.3.10)

	zsh: redirection with no command

This was exactly my "complaint" - that this is sensible syntax, that
could have worked, but doesn't.


-- 
Nadav Har'El                        |    Wednesday, Nov 25 2009, 8 Kislev 5770
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |A cat has claws ending its paws. A
http://nadav.harel.org.il           |sentence has a pause ending its clause.

Re: Redirecting shell output to a pipe

[Joke de Buhr]

[-- Attachment #1: Type: Text/Plain, Size: 1045 bytes --]

On Wednesday 25 November 2009 07:48:01 Nadav Har'El wrote:
> On Tue, Nov 24, 2009, Peter Miller wrote about "Re: Redirecting shell 
output to a pipe":
> >...
> >
> > > Isn't it natural to assume that in the same fashion, you should
> > > also be able to redirect the scripts output to a pipe? E.g., a very
> > > useful idiom could have been
> > >
> > > 	exec | tee filename
> >
> > exec |& tee filename

you should replace exec with a command. NOT the "exec" builtin. using the 
builtin exec command doesn't make much sense in this context:

RIGHT:
    echo "Hello Pipe!" |& tee filename
or
    echo "Hello Pipe!" 2>&1 | tee filename


  WRONG:    exec | tee filename
  WRONG:    exec |& tee filename
  USELESS:  exec echo "Hello Pipe!" | tee filename
  USELESS:  exec echo "Hello Pipe!" |& tee filename



> 
> Does this work for you? I get (on zsh 4.3.10)
> 
> 	zsh: redirection with no command
> 
> This was exactly my "complaint" - that this is sensible syntax, that
> could have worked, but doesn't.
> 

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 835 bytes --]

Re: Redirecting shell output to a pipe

[Nadav Har'El]

Hi,

On Wed, Nov 25, 2009, Joke de Buhr wrote about "Re: Redirecting shell output to a pipe":
> > > exec |& tee filename
> 
> you should replace exec with a command. NOT the "exec" builtin. using the 
> builtin exec command doesn't make much sense in this context:

I think there was a misunderstanding here. When I used the word "exec" I
didn't mean it as a replacement for any executable, but rather the specific
keyword "exec". I know all about the syntax for redirection of ordinary
commands (I've been using Unix shells for the past 25 years ;-)).
What I was asking was not about redirecting an invidual command, but rather
redirecting all the shell's output, from now on. The syntax to do that
for redirection to a file is:

	exec > filename

Just like that - with the keyword "exec". It's a bizarre syntax, I admit,
but one that has existed for decades (it is not a zsh-specific feature).
My question was how do you do that - i.e., redirection of all the shell's
output - but not to a file but rather a command, e.g., the command
"tee filename". I was hoping that the following would work:

	exec | tee filename

would work, but it doesn't, and I was wondering if anybody knew why - is
this an oversight or deliberate. I also gave a working, but way too difficult
alternative to achieve the same thing using a coprocess.

Thanks,
Nadav.

-- 
Nadav Har'El                        |    Wednesday, Nov 25 2009, 8 Kislev 5770
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |:(){ :|:&};: # DANGER: DO NOT run this,
http://nadav.harel.org.il           |unless you REALLY know what you're doing!

RE: Redirecting shell output to a pipe

[Norman.Azadian]

What you want is typescript.

NHA
---
Norman H. Azadian   SCS-NIT-DEV-PCI-FUF-NSP-SIA (EXT)
Swisscom AG      Zentweg 9      CH-3006 Ostermundigen
+41 31 342 2141           norman.azadian@swisscom.com


-----Original Message-----
From: Nadav Har'El [mailto:nyh@math.technion.ac.il] 
Sent: Wednesday, November 25, 2009 2:45 PM
To: Joke de Buhr
Cc: zsh-users@zsh.org
Subject: Re: Redirecting shell output to a pipe

Hi,

On Wed, Nov 25, 2009, Joke de Buhr wrote about "Re: Redirecting shell output to a pipe":
> > > exec |& tee filename
> 
> you should replace exec with a command. NOT the "exec" builtin. using the 
> builtin exec command doesn't make much sense in this context:

I think there was a misunderstanding here. When I used the word "exec" I
didn't mean it as a replacement for any executable, but rather the specific
keyword "exec". I know all about the syntax for redirection of ordinary
commands (I've been using Unix shells for the past 25 years ;-)).
What I was asking was not about redirecting an invidual command, but rather
redirecting all the shell's output, from now on. The syntax to do that
for redirection to a file is:

	exec > filename

Just like that - with the keyword "exec". It's a bizarre syntax, I admit,
but one that has existed for decades (it is not a zsh-specific feature).
My question was how do you do that - i.e., redirection of all the shell's
output - but not to a file but rather a command, e.g., the command
"tee filename". I was hoping that the following would work:

	exec | tee filename

would work, but it doesn't, and I was wondering if anybody knew why - is
this an oversight or deliberate. I also gave a working, but way too difficult
alternative to achieve the same thing using a coprocess.

Thanks,
Nadav.

-- 
Nadav Har'El                        |    Wednesday, Nov 25 2009, 8 Kislev 5770
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |:(){ :|:&};: # DANGER: DO NOT run this,
http://nadav.harel.org.il           |unless you REALLY know what you're doing!

Re: Redirecting shell output to a pipe

[Peter Stephenson]

On Wed, 25 Nov 2009 08:48:01 +0200
"Nadav Har'El" <nyh@math.technion.ac.il> wrote:
> On Tue, Nov 24, 2009, Peter Miller wrote about "Re: Redirecting shell output to a pipe":
> >...
> > > Isn't it natural to assume that in the same fashion, you should also
> > > be able to redirect the scripts output to a pipe? E.g., a very useful
> > > idiom could have been
> > >
> > > 	exec | tee filename
> > 
> > exec |& tee filename
> 
> Does this work for you? I get (on zsh 4.3.10)
> 
> 	zsh: redirection with no command
> 
> This was exactly my "complaint" - that this is sensible syntax, that
> could have worked, but doesn't.

I think this does the wrong thing: it uses NULLCMD, i.e.  executes "exec
cat | tee filename" (or otherwise, depending what $NULLCMD is), which isn't
what you want.  If you don't use a NULLCMD this fails entirely.  Adding
2>&1 doesn't change this.

Actually, I don't think this would ever work.  You need to understand what
a pipeline means to the shell: it means create a pipe, then start the
different parts of the pipeline, then wait for them to finish.  What you
want is beyond the limits of expression of the standard pipeline syntax:
you want it to say create this pipeline and leave it running as a separate
process with output redirected.

Zsh does have a syntax for exactly that:

exec > >(tee filename)

That might do what you want.

Note the "tee" process is asynchronous with respect to the rest of the
script, i.e. the shell won't wait for it to exit before the script exits.
That's normally the case with pipelines; the second process waits for the
first to exit.  I'm noting it specially here because it's perhaps less
clear that the shell is becoming the first part of a pipeline over which it
has no overall control, i.e. that it's different from running a pipeline
entirely within the shell.  But it may already be obvious that's the case.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom

Re: Redirecting shell output to a pipe

[Nadav Har'El]

On Wed, Nov 25, 2009, Peter Stephenson wrote about "Re: Redirecting shell output to a pipe":
>...
> > > > be able to redirect the scripts output to a pipe? E.g., a very useful
> > > > idiom could have been
> > > >
> > > > 	exec | tee filename
>...
> 
> different parts of the pipeline, then wait for them to finish.  What you
> want is beyond the limits of expression of the standard pipeline syntax:
> you want it to say create this pipeline and leave it running as a separate
> process with output redirected.
> 
> Zsh does have a syntax for exactly that:
> 
> exec > >(tee filename)
> 
> That might do what you want.

Beautiful!

> Note the "tee" process is asynchronous with respect to the rest of the
> script, i.e. the shell won't wait for it to exit before the script exits.
> That's normally the case with pipelines; the second process waits for the
> first to exit.  I'm noting it specially here because it's perhaps less
> clear that the shell is becoming the first part of a pipeline over which it
> has no overall control, i.e. that it's different from running a pipeline
> entirely within the shell.  But it may already be obvious that's the case.

Yes, my "coproc"-based solution also suffered from a related problem -
sometimes you lose the last line of output because the shell exited
(and killed the tee process) before it had the time to do its output.
I wonder, though, here, whether the shell will actually kill the tee or will
it continue to live forever (I didn't test).

Thanks,
Nadav.

-- 
Nadav Har'El                        |    Wednesday, Nov 25 2009, 8 Kislev 5770
nyh@math.technion.ac.il             |-----------------------------------------
Phone +972-523-790466, ICQ 13349191 |Attention: There will be a rain dance
http://nadav.harel.org.il           |Friday night, weather permitting.

Re: Redirecting shell output to a pipe

[Peter Stephenson]

"Nadav Har'El" wrote:
> > exec > >(tee filename)
>
> I wonder, though, here, whether the shell will actually kill the tee or will
> it continue to live forever (I didn't test).

The shell won't kill the tee, but it should get end-of-file on the pipe
when the script exits.  Basically, seen from outside it should behave as
a normal pipe: tee will exit the way it always does in a pipeline.  It's
only from inside the shell it looks a bit non-standard.

-- 
Peter Stephenson <pws@csr.com>            Software Engineer
Tel: +44 (0)1223 692070                   Cambridge Silicon Radio Limited
Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, UK


Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom