Mixing and multiple redirection of stderr/stdout

Mixing and multiple redirection of stderr/stdout

[Michael Lehn]

Hi,

I am wondering whether I can achieve the following with zsh:

Some script produces output on stdout and stderr, now I want

(1) redirect the sync-mix of stdout and stderr to some file file_mix
(2) redirect stdout to file_out
(3) redirect stderr to file_err

After reading the zsh-lovers man page I thought that it is possible.
However, I even fail when I try to achieve only (1) and (3) via

	./script > file_mix 2>&1 2>file_err

So I guess I completely did not get the point ... Could anybody
help me on this?

Michael

Re: Mixing and multiple redirection of stderr/stdout

[Bart Schaefer]

On May 18,  3:25am, Michael Lehn wrote:
}
} Some script produces output on stdout and stderr, now I want
} 
} (1) redirect the sync-mix of stdout and stderr to some file file_mix
} (2) redirect stdout to file_out
} (3) redirect stderr to file_err
} 
} However, I even fail when I try to achieve only (1) and (3) via
} 
} 	./script > file_mix 2>&1 2>file_err

This will only work if you have setopt MULT_IOS in effect, but if it
is, then the above should be correct.  Let's set up a little test
case so it's easier to see:

torch% [[ -o multios ]] && echo yes
yes

OK, we have multios.  Next we need a script that produces output on
both streams.

torch% function script { print STDOUT; print -u2 STDERR }

Then some "files" we can redirect into that will tell us what stream
the file is actually seeing.

torch% alias -g file_err='>(sed s/^/file_err:/)'
torch% alias -g file_out='>(sed s/^/file_out:/)'
torch% alias -g file_mix='>(sed s/^/file_mix:/)'

And now your (1) and (3) test:

torch% script >file_mix 2>&1 2>file_err          
file_mix:STDOUT
file_mix:STDERR
file_err:STDERR

The order of the redirections is important.  If we just throw >file_out
onto the end:

torch% script >file_mix 2>&1 2>file_err >file_out
file_mix:STDERR
file_out:STDOUT
file_err:STDERR

Here, 1 (stdout) is pointed at file_mix and then descriptor 2 (stderr)
is made a duplicate of descriptor 1; but then 1 is redirected again to
file_out leaving only 2 pointing at file_mix.  In effect "2>&1" turns
off the multio state for 1, and until *after* another redirection
like ">file_out", descriptor 1 is not again eligible for splitting
into more than one stream.

The non-obvious trick here is that you can re-enable the multio state
by redirecting a descriptor TO ITSELF:

torch% script >file_mix 2>&1 2>file_err 1>&1 >file_out
file_mix:STDERR
file_mix:STDOUT
file_err:STDERR
file_out:STDOUT

And there you have what you want.  However, it needn't be that ugly.

The next trick is that zsh has a shorthand operator ">&" (no numbers
around it) that means "redirect both stdout and stderr at the same
time".  This is done without destroying one of the original streams
in the way that happens when you use "2>&1".  So:

torch% script >&file_mix >file_out 2>file_err
file_out:STDOUT
file_mix:STDOUT
file_mix:STDERR
file_err:STDERR

And there you are.

-- 

Re: Mixing and multiple redirection of stderr/stdout

[Aaron Davies]

On Tuesday, May 17, 2011, Bart Schaefer <schaefer@brasslantern.com> wrote:

> The next trick is that zsh has a shorthand operator ">&" (no numbers
> around it) that means "redirect both stdout and stderr at the same
> time".  This is done without destroying one of the original streams
> in the way that happens when you use "2>&1".  So:
>
> torch% script >&file_mix >file_out 2>file_err
> file_out:STDOUT
> file_mix:STDOUT
> file_mix:STDERR
> file_err:STDERR
>
> And there you are.

Relatedly, can these (or any other) redirections guarantee the same
interleaving of stdout and stderr that would be produced on the
console with no redirections? I've often seen (mostly in other shells,
iirc) that a process which would have, e.g., alternating prints to out
and err, is captured to file as one long block of out followed by one
long block of err.

-- 
Aaron Davies
aaron.davies@gmail.com

Re: Mixing and multiple redirection of stderr/stdout

[Michael Lehn]

[-- Attachment #1: Type: text/plain, Size: 3402 bytes --]

Am 18.05.2011 um 07:13 schrieb Aaron Davies:

> On Tuesday, May 17, 2011, Bart Schaefer <schaefer@brasslantern.com> wrote:
> 
>> The next trick is that zsh has a shorthand operator ">&" (no numbers
>> around it) that means "redirect both stdout and stderr at the same
>> time".  This is done without destroying one of the original streams
>> in the way that happens when you use "2>&1".  So:
>> 
>> torch% script >&file_mix >file_out 2>file_err
>> file_out:STDOUT
>> file_mix:STDOUT
>> file_mix:STDERR
>> file_err:STDERR
>> 
>> And there you are.
> 
> Relatedly, can these (or any other) redirections guarantee the same
> interleaving of stdout and stderr that would be produced on the
> console with no redirections? I've often seen (mostly in other shells,
> iirc) that a process which would have, e.g., alternating prints to out
> and err, is captured to file as one long block of out followed by one
> long block of err.

Unfortunately I exactly encounter this problem now.  I am trying to apply
the trick Bart described as follows:

1) My example scripts echos some text to standard output.  The script
itself is a shell script which runs in verbose-mode.  Hence, every command
gets echoed to standard error before it gets executed:

----------------------------------------------------------------
zsh% cat script.sh 
#! /bin/zsh -v

echo 'Hello world!'
echo 'Hello world!'
----------------------------------------------------------------

2) If I just run the script I get the following output (where stdout and stderr
are mixed):

----------------------------------------------------------------
zsh% ./script.sh 
# system-wide environment settings for zsh(1)
if [ -x /usr/libexec/path_helper ]; then
	eval `/usr/libexec/path_helper -s`
fi
#! /bin/zsh -v

echo 'Hello world!'
Hello world!
echo 'Hello world!'
Hello world!
----------------------------------------------------------------

3) However, when I try mixing stdout&stderr and in addition try to redirect
stdout to a file I get random results in "file_mix".  For example   

----------------------------------------------------------------
zsh% ./script.sh >&file_mix >file_out           
zsh% cat file_mix 
# system-wide environment settings for zsh(1)
if [ -x /usr/libexec/path_helper ]; then
	eval `/usr/libexec/path_helper -s`
fi
#! /bin/zsh -v

echo 'Hello world!'
echo 'Hello world!'
Hello world!
Hello world!
----------------------------------------------------------------

This is just like when you would mix stdout and stderr by

	./script.sh >file_mix 2>file_mix

In this case this random behavior is clear.  We open the
file "file_mix" twice and therefore have two independent
entries in the open file table (one associated with stdout,
the other with stderr).

When one mixes the output by

	./script.sh >file_mix 2>&1

we only have one entry on the open file table for "file_mix"
and two file descriptors point the this entry.  Hence, the
outputs are mixed the right way (as they would be displayed
on the console).

So can something like my "mixing stdout&stderr but also redirect
stdout, stderr separately to files" work at all?  The longer I think
about it the more I am confused how it could be realized technically.
I guess it can not be done by using "open, close, dup and co" internally.
Or am I wrong?


Michael

Re: Mixing and multiple redirection of stderr/stdout

[Bart Schaefer]

On May 18, 11:18am, Michael Lehn wrote:
} 
} Am 18.05.2011 um 07:13 schrieb Aaron Davies:
} 
} > Relatedly, can these (or any other) redirections guarantee the
} > same interleaving of stdout and stderr that would be produced on
} > the console with no redirections? I've often seen (mostly in other
} > shells, iirc) that a process which would have, e.g., alternating
} > prints to out and err, is captured to file as one long block of out
} > followed by one long block of err.
}
} So can something like my "mixing stdout&stderr but also redirect
} stdout, stderr separately to files" work at all? The longer I think
} about it the more I am confused how it could be realized technically.
} I guess it can not be done by using "open, close, dup and co"
} internally. Or am I wrong?

In fact under the covers zsh is arranging something similar to

    exec 3>file_mix
    { script.sh |
      tee -a /proc/fd/3 |
      cat > file_out } 2>&1 |
     tee -a /proc/fd/3 |
     cat > file.err

so there is pipe buffering introduced that can affect the order of the
output in the mixed file even though the file is opened only once.

Even with >file_mix 2>&1 though, the order of output in the file can
be different from the order of output to the terminal because the
default OS buffering strategy for file descriptors may differ based
on the kind of device being written.  The introduction of extra pipes
just exaggerates this effect.

On the other hand zsh's internal tee/cat analogs are optimized for
efficiency rather than for preservation of ordering.  They could be
redone using select() and non-blocking descriptors to create a much
closer approximation of the interleaved write.

Re: Mixing and multiple redirection of stderr/stdout

[Michael Lehn]

Am 18.05.2011 um 16:26 schrieb Bart Schaefer:

> On May 18, 11:18am, Michael Lehn wrote:
> } 
> } Am 18.05.2011 um 07:13 schrieb Aaron Davies:
> } 
> } > Relatedly, can these (or any other) redirections guarantee the
> } > same interleaving of stdout and stderr that would be produced on
> } > the console with no redirections? I've often seen (mostly in other
> } > shells, iirc) that a process which would have, e.g., alternating
> } > prints to out and err, is captured to file as one long block of out
> } > followed by one long block of err.
> }
> } So can something like my "mixing stdout&stderr but also redirect
> } stdout, stderr separately to files" work at all? The longer I think
> } about it the more I am confused how it could be realized technically.
> } I guess it can not be done by using "open, close, dup and co"
> } internally. Or am I wrong?
> 
> In fact under the covers zsh is arranging something similar to
> 
>   exec 3>file_mix
>   { script.sh |
>     tee -a /proc/fd/3 |
>     cat > file_out } 2>&1 |
>    tee -a /proc/fd/3 |
>    cat > file.err
> 
> so there is pipe buffering introduced that can affect the order of the
> output in the mixed file even though the file is opened only once.

Thanks for clearing this up!  All this redirecting remains uncertain if
one does not have a glue about its internal realization.  

> 
> Even with >file_mix 2>&1 though, the order of output in the file can
> be different from the order of output to the terminal because the
> default OS buffering strategy for file descriptors may differ based
> on the kind of device being written.  The introduction of extra pipes
> just exaggerates this effect.

Yes, that explains the effect!

> 
> On the other hand zsh's internal tee/cat analogs are optimized for
> efficiency rather than for preservation of ordering.  They could be
> redone using select() and non-blocking descriptors to create a much
> closer approximation of the interleaved write.
> 

Right.  So the simplest solution would be writing a small C-programm
doing this.  I understand that a general purpose tool like a shell has
different priorities.

Thanks for your advice!

Michael

Re: Mixing and multiple redirection of stderr/stdout

[Aaron Davies]

On Wednesday, May 18, 2011, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On May 18, 11:18am, Michael Lehn wrote:
> }
> } Am 18.05.2011 um 07:13 schrieb Aaron Davies:
> }
> } > Relatedly, can these (or any other) redirections guarantee the
> } > same interleaving of stdout and stderr that would be produced on
> } > the console with no redirections? I've often seen (mostly in other
> } > shells, iirc) that a process which would have, e.g., alternating
> } > prints to out and err, is captured to file as one long block of out
> } > followed by one long block of err.
> }
> } So can something like my "mixing stdout&stderr but also redirect
> } stdout, stderr separately to files" work at all? The longer I think
> } about it the more I am confused how it could be realized technically.
> } I guess it can not be done by using "open, close, dup and co"
> } internally. Or am I wrong?
>
> In fact under the covers zsh is arranging something similar to
>
>     exec 3>file_mix
>     { script.sh |
>       tee -a /proc/fd/3 |
>       cat > file_out } 2>&1 |
>      tee -a /proc/fd/3 |
>      cat > file.err
>
> so there is pipe buffering introduced that can affect the order of the
> output in the mixed file even though the file is opened only once.
>
> Even with >file_mix 2>&1 though, the order of output in the file can
> be different from the order of output to the terminal because the
> default OS buffering strategy for file descriptors may differ based
> on the kind of device being written.  The introduction of extra pipes
> just exaggerates this effect.
>
> On the other hand zsh's internal tee/cat analogs are optimized for
> efficiency rather than for preservation of ordering.  They could be
> redone using select() and non-blocking descriptors to create a much
> closer approximation of the interleaved write.

I guess for now the best option is to capture at some other layer
(screen, the terminal, etc....)

-- 
Aaron Davies
aaron.davies@gmail.com