process substitution bug with set -e?

process substitution bug with set -e?

[Vincent Lefevre]

The zshexpn(1) man page of zsh 5.0.2 says:

  There  is  an additional problem with >(process); when this is attached
  to an external command, the parent shell does not wait for  process  to
  finish  and  hence  an immediately following command cannot rely on the
  results being complete.  The problem  and  solution  are  the  same  as
  described  in the section MULTIOS in zshmisc(1).  Hence in a simplified
  version of the example above:

         paste <(cut -f1 file1) <(cut -f3 file2) > >(process)

  (note that no MULTIOS are involved), process will be run asynchronously
  as far as the parent shell is concerned.  The workaround is:

         { paste <(cut -f1 file1) <(cut -f3 file2) } > >(process)

  The  extra  processes here are spawned from the parent shell which will
  wait for their completion.

Now, consider the following script:

#!/usr/bin/env zsh
set -e
{ /bin/cp } 2>>(sleep 1; cat -n)

Due to /bin/cp failure and the "set -e", the parent shell exits
immediately, without waiting for the extra processes:

ypig% ./zsh-procsubst
ypig%      1    /bin/cp: missing file operand
     2  Try '/bin/cp --help' for more information.

(tested under Debian/unstable).

Shouldn't the parent shell wait in this case?

If this is the expected behavior, the man page should be fixed,
and possibly give another workaround in the case of set -e.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: process substitution bug with set -e?

[Peter Stephenson]

On Mon, 14 Oct 2013 14:41:27 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> #!/usr/bin/env zsh
> set -e
> { /bin/cp } 2>>(sleep 1; cat -n)
> 
> Due to /bin/cp failure and the "set -e", the parent shell exits
> immediately, without waiting for the extra processes:
> 
> ypig% ./zsh-procsubst
> ypig%      1    /bin/cp: missing file operand
>      2  Try '/bin/cp --help' for more information.
> 
> (tested under Debian/unstable).
> 
> Shouldn't the parent shell wait in this case?

I may be thinking too naively here, but...

It's not clear to me this is wrong, anyway (apart from the lack of
documentation).  You're in a non-interactive shell with no job control
(it's possible to mix job control with ERR_EXIT although it seems rather
unnatural).  So the shell has no way cleanly to kill jobs associated
with it.  So it would have to wait until the sleep has finished (or any
other process, however long they took), and it doesn't necessarily know
they're going to exit --- which would be a bug in the script, but if
you've got ERR_EXIT set you probably want to avoid tickling script bugs
when that's in operation.  I think it could be made to wait, but
there's a reasonable argument that as it's already detected the failure
and you've asked it to exit on failure it should just do that.

I certainly don't claim this is a definitive answer.

pws

Re: process substitution bug with set -e?

[Vincent Lefevre]

On 2013-10-14 14:48:38 +0100, Peter Stephenson wrote:
> On Mon, 14 Oct 2013 14:41:27 +0200
> Vincent Lefevre <vincent@vinc17.net> wrote:
> > #!/usr/bin/env zsh
> > set -e
> > { /bin/cp } 2>>(sleep 1; cat -n)
> > 
> > Due to /bin/cp failure and the "set -e", the parent shell exits
> > immediately, without waiting for the extra processes:
> > 
> > ypig% ./zsh-procsubst
> > ypig%      1    /bin/cp: missing file operand
> >      2  Try '/bin/cp --help' for more information.
> > 
> > (tested under Debian/unstable).
> > 
> > Shouldn't the parent shell wait in this case?
> 
> I may be thinking too naively here, but...
> 
> It's not clear to me this is wrong, anyway (apart from the lack of
> documentation).  You're in a non-interactive shell with no job control
> (it's possible to mix job control with ERR_EXIT although it seems rather
> unnatural).  So the shell has no way cleanly to kill jobs associated
> with it.

I don't want them to be killed, on the contrary. In my real script,
the 2>>(...) is used to filter out informative messages and keep
real error messages. Killing this process would mean that error
messages would no longer be visible.

> So it would have to wait until the sleep has finished (or any
> other process, however long they took), and it doesn't necessarily know
> they're going to exit --- which would be a bug in the script, but if
> you've got ERR_EXIT set you probably want to avoid tickling script bugs
> when that's in operation.  I think it could be made to wait, but
> there's a reasonable argument that as it's already detected the failure
> and you've asked it to exit on failure it should just do that.
> 
> I certainly don't claim this is a definitive answer.

I can see that it has the same behavior as, for instance:

  { echo foo; exit } >>(sleep 1; cat -n)

Again, one may wonder whether the shell should exit immediately.
Is this clearly documented somewhere? I think this is the same
problem, and the precise behavior should be documented.

If the expected behavior is to exit immediately with "exit" or due
to a non-zero exit status with "set -e", then there should be a way
to behave as if the closing } were reached (instead of exiting
immediately). With EXIT and ZERR traps?

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: process substitution bug with set -e?

[Bart Schaefer]

On Oct 14,  5:08pm, Vincent Lefevre wrote:
} Subject: Re: process substitution bug with set -e?
}
} On 2013-10-14 14:48:38 +0100, Peter Stephenson wrote:
} > On Mon, 14 Oct 2013 14:41:27 +0200
} > Vincent Lefevre <vincent@vinc17.net> wrote:
} > > #!/usr/bin/env zsh
} > > set -e
} > > { /bin/cp } 2>>(sleep 1; cat -n)
} > > 
} > > Due to /bin/cp failure and the "set -e", the parent shell exits
} > > immediately, without waiting for the extra processes:
} > > 
} > > Shouldn't the parent shell wait in this case?
} > 
} > It's not clear to me this is wrong, anyway (apart from the lack of
} > documentation).  You're in a non-interactive shell with no job control
} > (it's possible to mix job control with ERR_EXIT although it seems rather
} > unnatural).

I tried the above with "setopt MONITOR" and that doesn't have any effect;
ERR_EXIT always wins, and MONITOR isn't necessary if ERR_EXIT is not set,
even if the shell is not interactive.

By comparison, if that command is run with "zsh -c ..." instead of as a
script, zsh *never* waits for >>(...), regardless of other setopts.  This
seems to be a side-effect of the exec optimization which decides that,
because there are no further commands to execute after { /bin/cp }, the
parent can simply start that up and disappear.

That is,

    zsh -c '{ /bin/cp } 2>>(sleep 1; cat -n)'

always exits as soon as /bin/cp exits, whereas

    zsh -c '{ /bin/cp } 2>>(sleep 1; cat -n) ; :'

waits for "cat -n" so that it can execute ":" after it.  (Interestingly,
adding a TRAPZERR also induces -c to wait, I'm not sure why.)

} > you've got ERR_EXIT set you probably want to avoid tickling script bugs
} > when that's in operation.  I think it could be made to wait, but
} > there's a reasonable argument that as it's already detected the failure
} > and you've asked it to exit on failure it should just do that.
} 
} I can see that it has the same behavior as, for instance:
} 
}   { echo foo; exit } >>(sleep 1; cat -n)
} 
} Again, one may wonder whether the shell should exit immediately.
} Is this clearly documented somewhere?

In general exiting from the shell either kills or disowns all jobs.  The
example of >>(process) is sort of a magic special case; it has to be run
asynchronously to be of any use, but "logically" it's part of the whole
job that is redirected to it, so when there is more to happen after it
zsh automatically does a "wait" for the implicitly backgrounded subshell.
That doesn't override the general rule about exiting.

Whether this is "clearly documented" ...

Incidentally exiting also bails out of "always" constructs without
executing the final block.

} If the expected behavior is to exit immediately with "exit" or due
} to a non-zero exit status with "set -e", then there should be a way
} to behave as if the closing } were reached (instead of exiting
} immediately). With EXIT and ZERR traps?

AFAICT this "there should be a way" isn't reflected in any other shell.
"TRAPEXIT() { wait }" almost works, but blocks the parent forever. (?)

Re: process substitution bug with set -e?

[Peter Stephenson]

On Mon, 14 Oct 2013 17:08:45 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> I can see that it has the same behavior as, for instance:
> 
>   { echo foo; exit } >>(sleep 1; cat -n)
> 
> Again, one may wonder whether the shell should exit immediately.
> Is this clearly documented somewhere?

These interactions between different features are definitely not clearly
documented.  This formally quadratic problem  --- it's not actually as bad
as (features)**2, of course --- is still fairly horrific in the case of
zsh.

I note that with a subshell, in

  ( echo foo ) >>(sleep 10; cat -n)

the shell waits but in

  ( echo foo; exit ) >>(sleep 10; cat -n)

it doesn't.  So this must mean the logic for waiting is inside the
subshell.  This surprised me.  I don't know if this has implications for
what's going on in the case of { ... }.

pws

Re: process substitution bug with set -e?

[Bart Schaefer]

On Oct 14,  5:28pm, Peter Stephenson wrote:
}
} These interactions between different features are definitely not clearly
} documented.  This formally quadratic problem  --- it's not actually as bad
} as (features)**2, of course --- is still fairly horrific in the case of
} zsh.

I'd argue that it's actually worse than (features)**2, because you have to
determine not only whether feature X interacts with feature Y, but whether
the interaction of X with Y futher interacts with feature Z.  E.g. just for
this thread we had ERR_EXIT, MONITOR, and process substitution involved.

So we've pretty much decided on a case-by-case basis what to document and
what not to, but in this instance we've done enough legwork that it is
probably worth writing it down.

Re: process substitution bug with set -e?

[Peter Stephenson]

On Mon, 14 Oct 2013 10:47:49 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> So we've pretty much decided on a case-by-case basis what to document and
> what not to, but in this instance we've done enough legwork that it is
> probably worth writing it down.

What scope should be trying to cover here?  Simply document for process
substiution that if the code to which the substitution applies exits
early the shell won't wait for it to finish?  Or can we infer something
a bit wider?

pws

Re: process substitution bug with set -e?

[Bart Schaefer]

On Oct 15,  9:42am, Peter Stephenson wrote:
}
} What scope should be trying to cover here?  Simply document for process
} substiution that if the code to which the substitution applies exits
} early the shell won't wait for it to finish?  Or can we infer something
} a bit wider?

I think we should be able to say something more general about the behavior
of "exit" with respect to asynchronous jobs, and that ERR_EXIT has the
same behavior.  There's a little about this in the "Jobs" section but it
only references explicitly backgrounded jobs and the HUP signal/option,
not about implicitly asynchronous jobs.

For example, note that multios and process substitutions are "disowned"
in the sense that they don't get HUP'd (they also don't get TTOU'd, but
aren't able to read from the terminal as far as I can tell [I/O error]).

We could further explain the generality that any pair of jobs that act as
a reader and a writer -- multios, process substitutions, pipes, are there
more? -- have to be run in parallel to avoid deadlock, which means that
at least one of each such pair is "implicitly asynchronous" in the sense
I used above.  Yeah, that's a misuse of "implicit" because the user has
of course explicity written >>(...) or whatever, so maybe there's another
way to say it.

I suppose the tail end of the "Jobs" section is the right place to put
this, even though that's somewhat distant from the descriptions of the
syntax that evoke the behavior.

Re: process substitution bug with set -e?

[Peter Stephenson]

On Tue, 15 Oct 2013 07:29:47 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I think we should be able to say something more general about the behavior
> of "exit" with respect to asynchronous jobs, and that ERR_EXIT has the
> same behavior.  There's a little about this in the "Jobs" section but it
> only references explicitly backgrounded jobs and the HUP signal/option,
> not about implicitly asynchronous jobs.
> 
> For example, note that multios and process substitutions are "disowned"
> in the sense that they don't get HUP'd (they also don't get TTOU'd, but
> aren't able to read from the terminal as far as I can tell [I/O error]).

Please provide any further patches that seem a good idea on top of this.

diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index 2cc33d2..1d9fe68 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -595,6 +595,11 @@ is specified, use the exit status from the last command executed.
 pindex(IGNORE_EOF, use of)
 An EOF condition will also cause the shell to exit, unless
 the tt(IGNORE_EOF) option is set.
+
+See notes at the end of
+ifzman(the section JOBS in in zmanref(zshmisc))\
+ifnzman(noderef(Jobs & Signals)) for some possibly unexpected interactions
+of the tt(exit) command with jobs.
 )
 findex(export)
 item(tt(export) [ var(name)[tt(=)var(value)] ... ])(
diff --git a/Doc/Zsh/jobs.yo b/Doc/Zsh/jobs.yo
index 3baf77f..d939501 100644
--- a/Doc/Zsh/jobs.yo
+++ b/Doc/Zsh/jobs.yo
@@ -115,3 +115,18 @@ The shell itself always ignores the tt(QUIT) signal.
 Otherwise, signals have the values
 inherited by the shell from its parent
 (but see the tt(TRAP)var(NAL) special functions in noderef(Functions)).
+
+cindex(exiting shell, and asynchronous jobs)
+cindex(asynchronous jobs, and exiting shell)
+cindex(jobs, asynchronous, and exiting shell)
+Certain jobs are run asynchronously by the shell other than those
+explicitly put into the background; even in cases where the shell
+would usually wait for such jobs, an explicit tt(exit) command
+or exit due to the option tt(ERR_EXIT) will cause the shell to
+exit without waiting.  Examples of such asynchronous jobs are
+process substitution, see
+ifzman(the section PROCESS SUBSTITUTION in the zmanref(zshexpn) manual page)\
+ifnzman(noderef(Process Substitution)), and the handler processes for
+multios, see
+ifzman(the section MULTIOS in the zmanref(zshmisc) manual page)\
+ifnzman(the section Multios in noderef(Redirection)).
diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo
index 9055215..3c6ea63 100644
--- a/Doc/Zsh/options.yo
+++ b/Doc/Zsh/options.yo
@@ -1553,6 +1553,11 @@ the trap.  If the option tt(DEBUG_BEFORE_CMD) is set,
 as it is by default, and the option tt(ERR_EXIT) is found to have been set
 on exit, then the command for which the tt(DEBUG) trap is being executed is
 skipped.  The option is restored after the trap exits.
+
+Exiting due to tt(ERR_EXIT) has certain interactions with asynchronous
+jobs noted in
+ifzman(the section JOBS in in zmanref(zshmisc))\
+ifnzman(noderef(Jobs & Signals)).
 )
 pindex(ERR_RETURN)
 pindex(NO_ERR_RETURN)

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/