coproc fd leak leading to possible deadlock

coproc fd leak leading to possible deadlock

[Stephane Chazelas]

When we run an external command as a coproc, all is fine, but if
we run a subshell, that subshell ends up having open fds for
both ends of both pipes, leading to potential deadlocks:

~$ exec zsh -f
sc% coproc {sleep 100; sleep 100}
[1] 14488
sc% ps -H
  PID TTY          TIME CMD
14484 pts/6    00:00:00 zsh
14488 pts/6    00:00:00   zsh
14489 pts/6    00:00:00     sleep
14490 pts/6    00:00:00   ps
sc% lsof -ad0-99 -p $!
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
zsh     14488 chazelas    0r  FIFO    0,8      0t0 294217 pipe
zsh     14488 chazelas    1w  FIFO    0,8      0t0 294216 pipe
zsh     14488 chazelas    2u   CHR  136,6      0t0      9 /dev/pts/6
zsh     14488 chazelas   10u   CHR  136,6      0t0      9 /dev/pts/6
zsh     14488 chazelas   11r  FIFO    0,8      0t0 294216 pipe
zsh     14488 chazelas   14w  FIFO    0,8      0t0 294217 pipe


Above, you see the lsof of the coproc. see how fd 11 is the
reading end of the pipe on fd 1 and fd 14 is the writing end of
the pipe on fd 0.

For an external command, it's OK:

sc% coproc sleep 20
[1] 14537
sc% lsof -ad0-99 -p $!
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
sleep   14537 chazelas    0r  FIFO    0,8      0t0 293743 pipe
sleep   14537 chazelas    1w  FIFO    0,8      0t0 293742 pipe
sleep   14537 chazelas    2u   CHR  136,1      0t0      4 /dev/pts/1


A work around is to do:

coproc sh -c 'sleep 100; exec sleep 100'

Side question: is it possible to close any of the coproc pipes?
>&p- or <&p- don't work.

If I run another coproc, that closes the current one AFAICT, so I
can do:

coproc cmd
exec {i}<&p {o}>&p
coproc :
exec {i}<&-

as a work around, but I thought there should be a simpler way.

Even when the coproc has terminated, the fds are not closed.

sc% coproc sleep 10
[1] 14594
sc% lsof -ad11-20 -p $$
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
zsh     12148 chazelas   11r  FIFO    0,8      0t0 295967 pipe
zsh     12148 chazelas   14w  FIFO    0,8      0t0 295968 pipe
sc%
[1]  + done       sleep 10
sc% lsof -ad11-20 -p $$
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
zsh     12148 chazelas   11r  FIFO    0,8      0t0 295967 pipe
zsh     12148 chazelas   14w  FIFO    0,8      0t0 295968 pipe

Having said that, zsh coproc is a lot more usable that bash's or ksh's. At
least the issues can easily be worked around.

-- 
Stephane

Re: coproc fd leak leading to possible deadlock

[Han Pingtian]

On Wed, Mar 06, 2013 at 10:53:41PM +0000, Stephane Chazelas wrote:
> Side question: is it possible to close any of the coproc pipes?
> >&p- or <&p- don't work.
> 
> If I run another coproc, that closes the current one AFAICT, so I
> can do:
> 
> coproc cmd
> exec {i}<&p {o}>&p
> coproc :
> exec {i}<&-
Interesting, I'm glad to know this. Thanks.
> 
> as a work around, but I thought there should be a simpler way.
> 
> Even when the coproc has terminated, the fds are not closed.
> 
> sc% coproc sleep 10
> [1] 14594
> sc% lsof -ad11-20 -p $$
> COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
> zsh     12148 chazelas   11r  FIFO    0,8      0t0 295967 pipe
> zsh     12148 chazelas   14w  FIFO    0,8      0t0 295968 pipe
> sc%
> [1]  + done       sleep 10
> sc% lsof -ad11-20 -p $$
> COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
> zsh     12148 chazelas   11r  FIFO    0,8      0t0 295967 pipe
> zsh     12148 chazelas   14w  FIFO    0,8      0t0 295968 pipe
> 
> Having said that, zsh coproc is a lot more usable that bash's or ksh's. At
> least the issues can easily be worked around.
> 
> -- 
> Stephane

Re: coproc fd leak leading to possible deadlock

[Bart Schaefer]

On Mar 6, 10:53pm, Stephane Chazelas wrote:
: Subject: coproc fd leak leading to possible deadlock
:
: When we run an external command as a coproc, all is fine, but if
: we run a subshell, that subshell ends up having open fds for
: both ends of both pipes, leading to potential deadlocks

Yes, I've been pointing out problems with coproc descriptor handling
for a while now (search zsh-worker mailing list archives).  So far I
have not found any issues with zsh's descriptor handling that are not
also shared by at least bash if not also ksh, so there hasn't been any
effort put into fixing things for a while.

: ~$ exec zsh -f
: sc% coproc {sleep 100; sleep 100}

The workaround for this one is to run a dummy coprocess *inside* the
coprocess:

% coproc { coproc : ; sleep 100; sleep 100 }

This should force closed the descriptors in the subshell that were
inherited from the parent shell, leaving the stdin/stdout connected to
the original pipes.

There's some discussion of why this happens (and why it's convoluted
to detect the situation) in the archives.  The short answer is that the
descriptors are close-on-exec [which is why "coproc sh ..." works] but
there's no exec with a subshell.