multios and unnecessary processes

multios and unnecessary processes

[Stephane Chazelas]

Hi, I would expect:

zsh -c 'lsof -ag $$ -d0-2,10-15 >&2 >&- >&2'

and 

zsh -c 'lsof -ag $$ -d0-2,10-15 >&2'

To work the same. However, in the first case, I get:

~$ zsh -c 'lsof -ag $$ -d0-2,10-15 >&2 >&- >&2'
~$ COMMAND  PID PGID     USER   FD   TYPE DEVICE SIZE  NODE NAME                                                                          16:39
lsof    7684 7684 chazelas    0u   CHR  136,6          8 /dev/pts/6
lsof    7684 7684 chazelas    1w  FIFO    0,7      21985 pipe
lsof    7684 7684 chazelas    2u   CHR  136,6          8 /dev/pts/6
zsh     7685 7684 chazelas   13u   CHR  136,6          8 /dev/pts/6
zsh     7685 7684 chazelas   14r  FIFO    0,7      21985 pipe

(note that my prompt is displayed before the result of lsof
because the output is written by that background zsh process).

In the second case, I get the expected behavior:

~$ zsh -c 'lsof -ag $$ -d0-2,10-15 >&2'
COMMAND  PID PGID     USER   FD   TYPE DEVICE SIZE NODE NAME
lsof    7702 7702 chazelas    0u   CHR  136,6         8 /dev/pts/6
lsof    7702 7702 chazelas    1u   CHR  136,6         8 /dev/pts/6
lsof    7702 7702 chazelas    2u   CHR  136,6         8 /dev/pts/6


I came accross this while trying to make some code independant
of the multios setting (hence the >&- to cancel the teed
redirection).

~$ zsh -f
% zmodload zsh/parameter
% echo $options[multios]
on

I'm not too sure it's a good idea to make multios the default.

-- 
Stéphane

Re: multios and unnecessary processes

[Bart Schaefer]

On Jan 9,  4:47pm, Stephane Chazelas wrote:
} Subject: multios and unnecessary processes
}
} zsh -c 'lsof -ag $$ -d0-2,10-15 >&2 >&- >&2'
} 
} and 
} 
} zsh -c 'lsof -ag $$ -d0-2,10-15 >&2'
} 
} To work the same.

Interesting.  For me, they *do* work the same *if* the command is run
from the shell prompt rather than from "zsh -c".  In fact, I can only
reproduce your results if I use a non-interactive shell.  If I put those
two commands in files and execute the files with "zsh -if" I get (in
both cases):

OMMAND   PID  PGRP     USER   FD   TYPE DEVICE SIZE   NODE NAME
zsh     22683 22683 schaefer    0u   CHR  136,8          10 /dev/pts/8
zsh     22683 22683 schaefer    1u   CHR  136,8          10 /dev/pts/8
zsh     22683 22683 schaefer    2u   CHR  136,8          10 /dev/pts/8
zsh     22683 22683 schaefer   10r   REG    3,1   36 159782 /tmp/lsof1
zsh     22683 22683 schaefer   11u   CHR  136,8          10 /dev/pts/8

But if I leave off the -i I get the extra pipe descriptors in the first
case.  I don't know offhand why multios would behave differently when
the shell is interactive.

} I came accross this while trying to make some code independant
} of the multios setting (hence the >&- to cancel the teed
} redirection).

I'm a little puzzled by that statement, because in your second example
there is no teed redirection.  It's only the doubled >&2 that creates
a teed redirection in the first place.

The right way to write multios-independent code is to wrap things in
curly braces, e.g.:  { lsof -ag $$ -d0-2,10-15 >&2 } >&2
 
} I'm not too sure it's a good idea to make multios the default.

If I had been in Paul Falstad's place 15 years or so ago, I'd have
thought about making the default value of multios follow the the initial
value of interactive; that is, having it on for interactive shells
unless explicitly turned off, and the reverse for non-interactive
shells.  Maybe he did think about it, but that's not what he decided.

Re: multios and unnecessary processes

[Stephane Chazelas]

On Sun, Jan 09, 2005 at 08:32:18PM +0000, Bart Schaefer wrote:
> On Jan 9,  4:47pm, Stephane Chazelas wrote:
> } Subject: multios and unnecessary processes
> }
> } zsh -c 'lsof -ag $$ -d0-2,10-15 >&2 >&- >&2'
> } 
> } and 
> } 
> } zsh -c 'lsof -ag $$ -d0-2,10-15 >&2'
> } 
> } To work the same.
> 
> Interesting.  For me, they *do* work the same *if* the command is run
> from the shell prompt rather than from "zsh -c".  In fact, I can only
> reproduce your results if I use a non-interactive shell.  If I put those
> two commands in files and execute the files with "zsh -if" I get (in
> both cases):
> 
> OMMAND   PID  PGRP     USER   FD   TYPE DEVICE SIZE   NODE NAME
> zsh     22683 22683 schaefer    0u   CHR  136,8          10 /dev/pts/8
> zsh     22683 22683 schaefer    1u   CHR  136,8          10 /dev/pts/8
> zsh     22683 22683 schaefer    2u   CHR  136,8          10 /dev/pts/8
> zsh     22683 22683 schaefer   10r   REG    3,1   36 159782 /tmp/lsof1
> zsh     22683 22683 schaefer   11u   CHR  136,8          10 /dev/pts/8
> 
> But if I leave off the -i I get the extra pipe descriptors in the first
> case.  I don't know offhand why multios would behave differently when
> the shell is interactive.
[...]

That's because of the -g option to lsof (process group $$, and
lsof is run in a different process group in interactive mode),
but you get the same behavior in interactive shells (you need to
give other options to lsof)

> } I came accross this while trying to make some code independant
> } of the multios setting (hence the >&- to cancel the teed
> } redirection).
> 
> I'm a little puzzled by that statement, because in your second example
> there is no teed redirection.  It's only the doubled >&2 that creates
> a teed redirection in the first place.
> 
> The right way to write multios-independent code is to wrap things in
> curly braces, e.g.:  { lsof -ag $$ -d0-2,10-15 >&2 } >&2

That was for:

{ cmd 2>&1 >&- >&3 3>&- | cmd2 3>&-; } 3>&-

(which can be written cmd 2> >(cmd2) on some systems)

Thanks for the

{ { cmd 2>&1 >&3 3>&-; } | cmd2 3>&-; } 3>&-

~$ zsh -c '{ { lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&-; } | tr a b 3>&-; } 3>&1'
COMMAND  PID PGID     USER   FD   TYPE DEVICE SIZE  NODE NAME
tr      4506 4506 chazelas    0u  FIFO    0,7      10296 pipe
tr      4506 4506 chazelas    1u   CHR  136,1          3 /dev/pts/1
tr      4506 4506 chazelas    2u   CHR  136,1          3 /dev/pts/1
zsh     4507 4506 chazelas    0u   CHR  136,1          3 /dev/pts/1
zsh     4507 4506 chazelas    1w  FIFO    0,7      10296 pipe
zsh     4507 4506 chazelas    2u   CHR  136,1          3 /dev/pts/1
zsh     4507 4506 chazelas   10u   CHR  136,1          3 /dev/pts/1
zsh     4507 4506 chazelas   11r   CHR    1,3       2892 /dev/null
lsof    4508 4506 chazelas    0u   CHR  136,1          3 /dev/pts/1
lsof    4508 4506 chazelas    1u   CHR  136,1          3 /dev/pts/1
lsof    4508 4506 chazelas    2w  FIFO    0,7      10296 pipe

But that's still one more process compared to:

~$ zsh -o nomultios -c '{ lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&- | tr a b 3>&-; } 3>&1'
COMMAND  PID PGID     USER   FD   TYPE DEVICE SIZE  NODE NAME
tr      4564 4564 chazelas    0u  FIFO    0,7      10603 pipe
tr      4564 4564 chazelas    1u   CHR  136,1          3 /dev/pts/1
tr      4564 4564 chazelas    2u   CHR  136,1          3 /dev/pts/1
lsof    4565 4564 chazelas    0u   CHR  136,1          3 /dev/pts/1
lsof    4565 4564 chazelas    1u   CHR  136,1          3 /dev/pts/1
lsof    4565 4564 chazelas    2w  FIFO    0,7      10603 pipe


-- 
Stéphane

Re: multios and unnecessary processes

[Bart Schaefer]

On Jan 10,  9:49am, Stephane Chazelas wrote:
} Subject: Re: multios and unnecessary processes
}
} > I don't know offhand why multios would behave differently when
} > the shell is interactive.
} [...]
} 
} That's because of the -g option to lsof (process group $$, and
} lsof is run in a different process group in interactive mode),

Ah, I should have thought of that.

} > The right way to write multios-independent code is to wrap things in
} > curly braces, e.g.:  { lsof -ag $$ -d0-2,10-15 >&2 } >&2
} 
} Thanks for the
} 
} { { cmd 2>&1 >&3 3>&-; } | cmd2 3>&-; } 3>&-
} 
} ~$ zsh -c '{ { lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&-; } | tr a b 3>&-; } 3>&1'
} But that's still one more process compared to:
} 
} ~$ zsh -o nomultios -c '{ lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&- | tr a b 3>&-; } 3>&1'

If the number of processes spawned is important, you have to know when
to use curly braces and when to force a subshell with parens.  Curly
braces imply that the parent zsh sticks around and waits, whereas a
subshell with parens can simply do an exec.

zsh -fc '{ ( lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&-; ) | tr a b 3>&-; } 3>&1'

COMMAND   PID  PGRP     USER   FD   TYPE DEVICE SIZE  NODE NAME
tr      26811 26811 schaefer    0r  FIFO    0,0      72995 pipe
tr      26811 26811 schaefer    1u   CHR  136,7          9 /dev/pts/7
tr      26811 26811 schaefer    2u   CHR  136,7          9 /dev/pts/7
lsof    26822 26811 schaefer    0u   CHR  136,7          9 /dev/pts/7
lsof    26822 26811 schaefer    1u   CHR  136,7          9 /dev/pts/7
lsof    26822 26811 schaefer    2w  FIFO    0,0      72995 pipe

Re: multios and unnecessary processes

[Stephane Chazelas]

On Mon, Jan 10, 2005 at 04:53:25PM +0000, Bart Schaefer wrote:
[...]
> } ~$ zsh -o nomultios -c '{ lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&- | tr a b 3>&-; } 3>&1'
> 
> If the number of processes spawned is important, you have to know when
> to use curly braces and when to force a subshell with parens.  Curly
> braces imply that the parent zsh sticks around and waits, whereas a
> subshell with parens can simply do an exec.
> 
> zsh -fc '{ ( lsof -ag $$ -d 0-2,10-15 2>&1 >&3 3>&-; ) | tr a b 3>&-; } 3>&1'
> 
> COMMAND   PID  PGRP     USER   FD   TYPE DEVICE SIZE  NODE NAME
> tr      26811 26811 schaefer    0r  FIFO    0,0      72995 pipe
> tr      26811 26811 schaefer    1u   CHR  136,7          9 /dev/pts/7
> tr      26811 26811 schaefer    2u   CHR  136,7          9 /dev/pts/7
> lsof    26822 26811 schaefer    0u   CHR  136,7          9 /dev/pts/7
> lsof    26822 26811 schaefer    1u   CHR  136,7          9 /dev/pts/7
> lsof    26822 26811 schaefer    2w  FIFO    0,0      72995 pipe

That makes sense but is not very intuitive given that (...) is
generally thought as the form spawning additional processes.

However to get back to my initial statement, don't you agree
it's a problem that

cmd >&2 >&- >&2

doesn't redirect stdout to stderr but to a pipe to a background
process that just echos the output to stderr?

That's not too big an issue. I was just trying to give some code
example in comp.unix.shell that would work the same for any shell
(including zsh with multios).

-- 
Stéphane

Re: multios and unnecessary processes

[Bart Schaefer]

On Jan 10,  5:11pm, Stephane Chazelas wrote:
} Subject: Re: multios and unnecessary processes
}
} However to get back to my initial statement, don't you agree
} it's a problem that
} 
} cmd >&2 >&- >&2
} 
} doesn't redirect stdout to stderr but to a pipe to a background
} process that just echos the output to stderr?

Yes, that could be considered a bug.  Suggested patch below.

But under what circumstances would you write that?

The only case where I can think offhand of that coming up is something
like this:

  original_cmd="lsof -ag $$ -d 0-2,10-15 >&2"
  # ... time and code passes ...
  modified_cmd="$original_cmd >&-"
  # ... more time and code passes ...
  finished_cmd="$modified_cmd >&2"
  eval $finished_cmd

Here's the patch.  This relies on the fact that mfds[] elements are heap
allocated; it also means that repeatedly opening and closing a descriptor
with MULTIOS on will repeatedly allocate and discard a multio structure,
whereas before it would re-use it even after closing.

Patch line numbers may be off because I've applied 20632 (PWS, I think
you could commit that, it seems to be fine).

Index: Src/exec.c
===================================================================
RCS file: /extra/cvsroot/zsh/zsh-4.0/Src/exec.c,v
retrieving revision 1.22
diff -c -r1.22 exec.c
--- exec.c	6 Dec 2004 16:51:18 -0000	1.22
+++ exec.c	10 Jan 2005 19:08:51 -0000
@@ -1487,6 +1488,7 @@
 	}
 	_exit(0);
     }
+    mfds[fd] = NULL;
 }
 
 /* close all the mnodes (failure) */

Re: multios and unnecessary processes

[Peter Stephenson]

Bart Schaefer wrote:
> Patch line numbers may be off because I've applied 20632 (PWS, I think
> you could commit that, it seems to be fine).

That's been in for a few days now.

pws


**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

Re: multios and unnecessary processes

[Stephane Chazelas]

On Mon, Jan 10, 2005 at 07:20:41PM +0000, Bart Schaefer wrote:
> On Jan 10,  5:11pm, Stephane Chazelas wrote:
> } Subject: Re: multios and unnecessary processes
> }
> } However to get back to my initial statement, don't you agree
> } it's a problem that
> } 
> } cmd >&2 >&- >&2
> } 
> } doesn't redirect stdout to stderr but to a pipe to a background
> } process that just echos the output to stderr?
> 
> Yes, that could be considered a bug.  Suggested patch below.
> 
> But under what circumstances would you write that?
[...]

Hi, thanks for the patch (I've not tested it yet).

As I said in a previous mail, I came accross that when trying to
cancel a multios:

{
cmd 2>&1 >&- >&3 | grep -v do-not-want-that-error-message
} 3>&1

Without >&-, cmd's stdout would have gone to the terminal and to
grep, and I just wanted cmd's stderr to go to grep.

At that time, I didn't know of:

(cmd 2>&1 >&3) | grep ...


-- 
Stéphane