* process substitution bug with set -e? @ 2013-10-14 12:41 Vincent Lefevre 2013-10-14 13:48 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Vincent Lefevre @ 2013-10-14 12:41 UTC (permalink / raw) To: zsh-users The zshexpn(1) man page of zsh 5.0.2 says: There is an additional problem with >(process); when this is attached to an external command, the parent shell does not wait for process to finish and hence an immediately following command cannot rely on the results being complete. The problem and solution are the same as described in the section MULTIOS in zshmisc(1). Hence in a simplified version of the example above: paste <(cut -f1 file1) <(cut -f3 file2) > >(process) (note that no MULTIOS are involved), process will be run asynchronously as far as the parent shell is concerned. The workaround is: { paste <(cut -f1 file1) <(cut -f3 file2) } > >(process) The extra processes here are spawned from the parent shell which will wait for their completion. Now, consider the following script: #!/usr/bin/env zsh set -e { /bin/cp } 2>>(sleep 1; cat -n) Due to /bin/cp failure and the "set -e", the parent shell exits immediately, without waiting for the extra processes: ypig% ./zsh-procsubst ypig% 1 /bin/cp: missing file operand 2 Try '/bin/cp --help' for more information. (tested under Debian/unstable). Shouldn't the parent shell wait in this case? If this is the expected behavior, the man page should be fixed, and possibly give another workaround in the case of set -e. -- Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-14 12:41 process substitution bug with set -e? Vincent Lefevre @ 2013-10-14 13:48 ` Peter Stephenson 2013-10-14 15:08 ` Vincent Lefevre 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-14 13:48 UTC (permalink / raw) To: zsh-users On Mon, 14 Oct 2013 14:41:27 +0200 Vincent Lefevre <vincent@vinc17.net> wrote: > #!/usr/bin/env zsh > set -e > { /bin/cp } 2>>(sleep 1; cat -n) > > Due to /bin/cp failure and the "set -e", the parent shell exits > immediately, without waiting for the extra processes: > > ypig% ./zsh-procsubst > ypig% 1 /bin/cp: missing file operand > 2 Try '/bin/cp --help' for more information. > > (tested under Debian/unstable). > > Shouldn't the parent shell wait in this case? I may be thinking too naively here, but... It's not clear to me this is wrong, anyway (apart from the lack of documentation). You're in a non-interactive shell with no job control (it's possible to mix job control with ERR_EXIT although it seems rather unnatural). So the shell has no way cleanly to kill jobs associated with it. So it would have to wait until the sleep has finished (or any other process, however long they took), and it doesn't necessarily know they're going to exit --- which would be a bug in the script, but if you've got ERR_EXIT set you probably want to avoid tickling script bugs when that's in operation. I think it could be made to wait, but there's a reasonable argument that as it's already detected the failure and you've asked it to exit on failure it should just do that. I certainly don't claim this is a definitive answer. pws ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-14 13:48 ` Peter Stephenson @ 2013-10-14 15:08 ` Vincent Lefevre 2013-10-14 16:23 ` Bart Schaefer 2013-10-14 16:28 ` Peter Stephenson 0 siblings, 2 replies; 9+ messages in thread From: Vincent Lefevre @ 2013-10-14 15:08 UTC (permalink / raw) To: zsh-users On 2013-10-14 14:48:38 +0100, Peter Stephenson wrote: > On Mon, 14 Oct 2013 14:41:27 +0200 > Vincent Lefevre <vincent@vinc17.net> wrote: > > #!/usr/bin/env zsh > > set -e > > { /bin/cp } 2>>(sleep 1; cat -n) > > > > Due to /bin/cp failure and the "set -e", the parent shell exits > > immediately, without waiting for the extra processes: > > > > ypig% ./zsh-procsubst > > ypig% 1 /bin/cp: missing file operand > > 2 Try '/bin/cp --help' for more information. > > > > (tested under Debian/unstable). > > > > Shouldn't the parent shell wait in this case? > > I may be thinking too naively here, but... > > It's not clear to me this is wrong, anyway (apart from the lack of > documentation). You're in a non-interactive shell with no job control > (it's possible to mix job control with ERR_EXIT although it seems rather > unnatural). So the shell has no way cleanly to kill jobs associated > with it. I don't want them to be killed, on the contrary. In my real script, the 2>>(...) is used to filter out informative messages and keep real error messages. Killing this process would mean that error messages would no longer be visible. > So it would have to wait until the sleep has finished (or any > other process, however long they took), and it doesn't necessarily know > they're going to exit --- which would be a bug in the script, but if > you've got ERR_EXIT set you probably want to avoid tickling script bugs > when that's in operation. I think it could be made to wait, but > there's a reasonable argument that as it's already detected the failure > and you've asked it to exit on failure it should just do that. > > I certainly don't claim this is a definitive answer. I can see that it has the same behavior as, for instance: { echo foo; exit } >>(sleep 1; cat -n) Again, one may wonder whether the shell should exit immediately. Is this clearly documented somewhere? I think this is the same problem, and the precise behavior should be documented. If the expected behavior is to exit immediately with "exit" or due to a non-zero exit status with "set -e", then there should be a way to behave as if the closing } were reached (instead of exiting immediately). With EXIT and ZERR traps? -- Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/> 100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/> Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-14 15:08 ` Vincent Lefevre @ 2013-10-14 16:23 ` Bart Schaefer 2013-10-14 16:28 ` Peter Stephenson 1 sibling, 0 replies; 9+ messages in thread From: Bart Schaefer @ 2013-10-14 16:23 UTC (permalink / raw) To: zsh-users On Oct 14, 5:08pm, Vincent Lefevre wrote: } Subject: Re: process substitution bug with set -e? } } On 2013-10-14 14:48:38 +0100, Peter Stephenson wrote: } > On Mon, 14 Oct 2013 14:41:27 +0200 } > Vincent Lefevre <vincent@vinc17.net> wrote: } > > #!/usr/bin/env zsh } > > set -e } > > { /bin/cp } 2>>(sleep 1; cat -n) } > > } > > Due to /bin/cp failure and the "set -e", the parent shell exits } > > immediately, without waiting for the extra processes: } > > } > > Shouldn't the parent shell wait in this case? } > } > It's not clear to me this is wrong, anyway (apart from the lack of } > documentation). You're in a non-interactive shell with no job control } > (it's possible to mix job control with ERR_EXIT although it seems rather } > unnatural). I tried the above with "setopt MONITOR" and that doesn't have any effect; ERR_EXIT always wins, and MONITOR isn't necessary if ERR_EXIT is not set, even if the shell is not interactive. By comparison, if that command is run with "zsh -c ..." instead of as a script, zsh *never* waits for >>(...), regardless of other setopts. This seems to be a side-effect of the exec optimization which decides that, because there are no further commands to execute after { /bin/cp }, the parent can simply start that up and disappear. That is, zsh -c '{ /bin/cp } 2>>(sleep 1; cat -n)' always exits as soon as /bin/cp exits, whereas zsh -c '{ /bin/cp } 2>>(sleep 1; cat -n) ; :' waits for "cat -n" so that it can execute ":" after it. (Interestingly, adding a TRAPZERR also induces -c to wait, I'm not sure why.) } > you've got ERR_EXIT set you probably want to avoid tickling script bugs } > when that's in operation. I think it could be made to wait, but } > there's a reasonable argument that as it's already detected the failure } > and you've asked it to exit on failure it should just do that. } } I can see that it has the same behavior as, for instance: } } { echo foo; exit } >>(sleep 1; cat -n) } } Again, one may wonder whether the shell should exit immediately. } Is this clearly documented somewhere? In general exiting from the shell either kills or disowns all jobs. The example of >>(process) is sort of a magic special case; it has to be run asynchronously to be of any use, but "logically" it's part of the whole job that is redirected to it, so when there is more to happen after it zsh automatically does a "wait" for the implicitly backgrounded subshell. That doesn't override the general rule about exiting. Whether this is "clearly documented" ... Incidentally exiting also bails out of "always" constructs without executing the final block. } If the expected behavior is to exit immediately with "exit" or due } to a non-zero exit status with "set -e", then there should be a way } to behave as if the closing } were reached (instead of exiting } immediately). With EXIT and ZERR traps? AFAICT this "there should be a way" isn't reflected in any other shell. "TRAPEXIT() { wait }" almost works, but blocks the parent forever. (?) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-14 15:08 ` Vincent Lefevre 2013-10-14 16:23 ` Bart Schaefer @ 2013-10-14 16:28 ` Peter Stephenson 2013-10-14 17:47 ` Bart Schaefer 1 sibling, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-14 16:28 UTC (permalink / raw) To: zsh-users On Mon, 14 Oct 2013 17:08:45 +0200 Vincent Lefevre <vincent@vinc17.net> wrote: > I can see that it has the same behavior as, for instance: > > { echo foo; exit } >>(sleep 1; cat -n) > > Again, one may wonder whether the shell should exit immediately. > Is this clearly documented somewhere? These interactions between different features are definitely not clearly documented. This formally quadratic problem --- it's not actually as bad as (features)**2, of course --- is still fairly horrific in the case of zsh. I note that with a subshell, in ( echo foo ) >>(sleep 10; cat -n) the shell waits but in ( echo foo; exit ) >>(sleep 10; cat -n) it doesn't. So this must mean the logic for waiting is inside the subshell. This surprised me. I don't know if this has implications for what's going on in the case of { ... }. pws ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-14 16:28 ` Peter Stephenson @ 2013-10-14 17:47 ` Bart Schaefer 2013-10-15 8:42 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2013-10-14 17:47 UTC (permalink / raw) To: zsh-users On Oct 14, 5:28pm, Peter Stephenson wrote: } } These interactions between different features are definitely not clearly } documented. This formally quadratic problem --- it's not actually as bad } as (features)**2, of course --- is still fairly horrific in the case of } zsh. I'd argue that it's actually worse than (features)**2, because you have to determine not only whether feature X interacts with feature Y, but whether the interaction of X with Y futher interacts with feature Z. E.g. just for this thread we had ERR_EXIT, MONITOR, and process substitution involved. So we've pretty much decided on a case-by-case basis what to document and what not to, but in this instance we've done enough legwork that it is probably worth writing it down. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-14 17:47 ` Bart Schaefer @ 2013-10-15 8:42 ` Peter Stephenson 2013-10-15 14:29 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-15 8:42 UTC (permalink / raw) To: zsh-users On Mon, 14 Oct 2013 10:47:49 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > So we've pretty much decided on a case-by-case basis what to document and > what not to, but in this instance we've done enough legwork that it is > probably worth writing it down. What scope should be trying to cover here? Simply document for process substiution that if the code to which the substitution applies exits early the shell won't wait for it to finish? Or can we infer something a bit wider? pws ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-15 8:42 ` Peter Stephenson @ 2013-10-15 14:29 ` Bart Schaefer 2013-10-16 17:53 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2013-10-15 14:29 UTC (permalink / raw) To: zsh-users On Oct 15, 9:42am, Peter Stephenson wrote: } } What scope should be trying to cover here? Simply document for process } substiution that if the code to which the substitution applies exits } early the shell won't wait for it to finish? Or can we infer something } a bit wider? I think we should be able to say something more general about the behavior of "exit" with respect to asynchronous jobs, and that ERR_EXIT has the same behavior. There's a little about this in the "Jobs" section but it only references explicitly backgrounded jobs and the HUP signal/option, not about implicitly asynchronous jobs. For example, note that multios and process substitutions are "disowned" in the sense that they don't get HUP'd (they also don't get TTOU'd, but aren't able to read from the terminal as far as I can tell [I/O error]). We could further explain the generality that any pair of jobs that act as a reader and a writer -- multios, process substitutions, pipes, are there more? -- have to be run in parallel to avoid deadlock, which means that at least one of each such pair is "implicitly asynchronous" in the sense I used above. Yeah, that's a misuse of "implicit" because the user has of course explicity written >>(...) or whatever, so maybe there's another way to say it. I suppose the tail end of the "Jobs" section is the right place to put this, even though that's somewhat distant from the descriptions of the syntax that evoke the behavior. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: process substitution bug with set -e? 2013-10-15 14:29 ` Bart Schaefer @ 2013-10-16 17:53 ` Peter Stephenson 0 siblings, 0 replies; 9+ messages in thread From: Peter Stephenson @ 2013-10-16 17:53 UTC (permalink / raw) To: zsh-users On Tue, 15 Oct 2013 07:29:47 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > I think we should be able to say something more general about the behavior > of "exit" with respect to asynchronous jobs, and that ERR_EXIT has the > same behavior. There's a little about this in the "Jobs" section but it > only references explicitly backgrounded jobs and the HUP signal/option, > not about implicitly asynchronous jobs. > > For example, note that multios and process substitutions are "disowned" > in the sense that they don't get HUP'd (they also don't get TTOU'd, but > aren't able to read from the terminal as far as I can tell [I/O error]). Please provide any further patches that seem a good idea on top of this. diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo index 2cc33d2..1d9fe68 100644 --- a/Doc/Zsh/builtins.yo +++ b/Doc/Zsh/builtins.yo @@ -595,6 +595,11 @@ is specified, use the exit status from the last command executed. pindex(IGNORE_EOF, use of) An EOF condition will also cause the shell to exit, unless the tt(IGNORE_EOF) option is set. + +See notes at the end of +ifzman(the section JOBS in in zmanref(zshmisc))\ +ifnzman(noderef(Jobs & Signals)) for some possibly unexpected interactions +of the tt(exit) command with jobs. ) findex(export) item(tt(export) [ var(name)[tt(=)var(value)] ... ])( diff --git a/Doc/Zsh/jobs.yo b/Doc/Zsh/jobs.yo index 3baf77f..d939501 100644 --- a/Doc/Zsh/jobs.yo +++ b/Doc/Zsh/jobs.yo @@ -115,3 +115,18 @@ The shell itself always ignores the tt(QUIT) signal. Otherwise, signals have the values inherited by the shell from its parent (but see the tt(TRAP)var(NAL) special functions in noderef(Functions)). + +cindex(exiting shell, and asynchronous jobs) +cindex(asynchronous jobs, and exiting shell) +cindex(jobs, asynchronous, and exiting shell) +Certain jobs are run asynchronously by the shell other than those +explicitly put into the background; even in cases where the shell +would usually wait for such jobs, an explicit tt(exit) command +or exit due to the option tt(ERR_EXIT) will cause the shell to +exit without waiting. Examples of such asynchronous jobs are +process substitution, see +ifzman(the section PROCESS SUBSTITUTION in the zmanref(zshexpn) manual page)\ +ifnzman(noderef(Process Substitution)), and the handler processes for +multios, see +ifzman(the section MULTIOS in the zmanref(zshmisc) manual page)\ +ifnzman(the section Multios in noderef(Redirection)). diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo index 9055215..3c6ea63 100644 --- a/Doc/Zsh/options.yo +++ b/Doc/Zsh/options.yo @@ -1553,6 +1553,11 @@ the trap. If the option tt(DEBUG_BEFORE_CMD) is set, as it is by default, and the option tt(ERR_EXIT) is found to have been set on exit, then the command for which the tt(DEBUG) trap is being executed is skipped. The option is restored after the trap exits. + +Exiting due to tt(ERR_EXIT) has certain interactions with asynchronous +jobs noted in +ifzman(the section JOBS in in zmanref(zshmisc))\ +ifnzman(noderef(Jobs & Signals)). ) pindex(ERR_RETURN) pindex(NO_ERR_RETURN) -- Peter Stephenson <p.w.stephenson@ntlworld.com> Web page now at http://homepage.ntlworld.com/p.w.stephenson/ ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-10-16 17:54 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-10-14 12:41 process substitution bug with set -e? Vincent Lefevre 2013-10-14 13:48 ` Peter Stephenson 2013-10-14 15:08 ` Vincent Lefevre 2013-10-14 16:23 ` Bart Schaefer 2013-10-14 16:28 ` Peter Stephenson 2013-10-14 17:47 ` Bart Schaefer 2013-10-15 8:42 ` Peter Stephenson 2013-10-15 14:29 ` Bart Schaefer 2013-10-16 17:53 ` Peter Stephenson
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).