* Re: No pipefail option? [not found] ` <20131005223159.25fea6a0@pws-pc.ntlworld.com> @ 2013-10-07 0:36 ` Bart Schaefer 2013-10-07 9:25 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2013-10-07 0:36 UTC (permalink / raw) To: zsh-workers [>workers] On Oct 5, 10:31pm, Peter Stephenson wrote: } } Indeed, if $pipestatus is working as advertised, it really ought to be } this simple... Unfortunately, $pipestatus is not working as advertised. See this thread from 2011: http://www.zsh.org/mla/workers/2011/msg01394.html In particular http://www.zsh.org/mla/workers/2011/msg01396.html http://www.zsh.org/mla/workers/2011/msg01470.html http://www.zsh.org/mla/workers/2011/msg01475.html http://www.zsh.org/mla/workers/2011/msg01476.html The patch mentined in msg01476 is here: http://www.zsh.org/mla/workers//2011/msg01472.html But it didn't resolve the issue, so I never committed it. The long and short of it is that pipestatus only works reliably when all the commands in the pipeline are external programs. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: No pipefail option? 2013-10-07 0:36 ` No pipefail option? Bart Schaefer @ 2013-10-07 9:25 ` Peter Stephenson 2013-10-07 14:40 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-07 9:25 UTC (permalink / raw) To: zsh-workers On Sun, 06 Oct 2013 17:36:21 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > [>workers] > > On Oct 5, 10:31pm, Peter Stephenson wrote: > } > } Indeed, if $pipestatus is working as advertised, it really ought to be > } this simple... > > Unfortunately, $pipestatus is not working as advertised. Looks that this might not be fatal for the most typical uses of PIPEFAIL, since I suspect if you've got a loop at the end of the pipe processing the results in the main shell you'd typically detect errors at that point. > See this thread from 2011: > > http://www.zsh.org/mla/workers/2011/msg01394.html > > In particular > > http://www.zsh.org/mla/workers/2011/msg01396.html > http://www.zsh.org/mla/workers/2011/msg01470.html > http://www.zsh.org/mla/workers/2011/msg01475.html > http://www.zsh.org/mla/workers/2011/msg01476.html > > The patch mentined in msg01476 is here: > > http://www.zsh.org/mla/workers//2011/msg01472.html > > But it didn't resolve the issue, so I never committed it. > > The long and short of it is that pipestatus only works reliably when > all the commands in the pipeline are external programs. It's not clear to me if (i) the problem is only relevant to the final stage in the pipeline (as we fork for others anyway) (ii) you need a complex command rather than a simple builtin to get confused, but presumably a complex command might be a function as there's nothing special about jobs run from inside a function (they are not encapsulated as far as job control is concerned). I haven't yet quite worked out why it's not possible to detect the point at which the job is finished and examine the state at that point, so I can't comment further, but I might look again and ask stupid questions. pws ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: No pipefail option? 2013-10-07 9:25 ` Peter Stephenson @ 2013-10-07 14:40 ` Bart Schaefer 2013-10-08 20:44 ` Shell job structure Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2013-10-07 14:40 UTC (permalink / raw) To: zsh-workers On Oct 7, 10:25am, Peter Stephenson wrote: } } I haven't yet quite worked out why it's not possible to detect the point } at which the job is finished and examine the state at that point, so I } can't comment further, but I might look again and ask stupid questions. I don't remember clearly all the details, but it's some combination of: - because external jobs can exit and be reaped in arbitrary order, even in a pipeline, the job table is used to keep track of which position in the array to update - jobs that run in the current shell don't have a complete job table entry [cf. all the gyrations to suspend/background a loop] and aren't "reaped" in the same code path as external jobs, so the wrong array position (or none at all) may get updated - the incorrect update depends on whether the external job exited AND got reaped before the current-shell job has completed, because of the way reaping updates the job table, so correctness is unpredictable - complex commands "in the current shell" may have external subjobs that need a separate pipestatus (this applies only at end of a pipeline) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Shell job structure 2013-10-07 14:40 ` Bart Schaefer @ 2013-10-08 20:44 ` Peter Stephenson 2013-10-25 5:02 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-08 20:44 UTC (permalink / raw) To: zsh-workers On Mon, 07 Oct 2013 07:40:49 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > - because external jobs can exit and be reaped in arbitrary order, even > in a pipeline, the job table is used to keep track of which position > in the array to update > > - jobs that run in the current shell don't have a complete job table entry > [cf. all the gyrations to suspend/background a loop] and aren't "reaped" > in the same code path as external jobs, so the wrong array position (or > none at all) may get updated > > - the incorrect update depends on whether the external job exited AND got > reaped before the current-shell job has completed, because of the way > reaping updates the job table, so correctness is unpredictable > > - complex commands "in the current shell" may have external subjobs that > need a separate pipestatus (this applies only at end of a pipeline) This may be pie in the sky, but probably what we need is for jobs to be hierarchical. A job at the top level would always be treated as separate from another job at the top level for the original purpose of job control but might include nested jobs, representing objects such as functions and chunks of pipelines, that might themselves have jobs representing external commands. Furthermore, a job would become a first class representation of shell state, so anything just involving shell builtins would have a job, removing the obfuscation in the current relationship between jobs and code execution. It would thus in principle allow better encapsulation of other shell state. The job table would become a list of top-level jobs, while the job structure would have pointers to nested jobs. We might get rid of the table all together and simply have a single pointer that got searched for top-level jobs the same way those jobs got searched for subjobs. If this was inefficient for searching we could hash PIDs and the like; if it was inefficient for memory management (but I don't see why it should be, particularly, compared with memory management for anything else) we could still have a pool of job structures. I think this means we'd have a status associated with a job, not just a process. How you got the job status would depend on the nature of the job. I think a job would be one of: - An external command with one main process and possibly auxiliary processes; the status of the job is just that of the process. This is sort of a degenerate case of a pipeline but I'm wondering it might be neater to make it just one part of a pipeline so auxiliary processes get tied to the right main process. This also makes an external command in a pipeline more similar to the case of a shell construct in a pipeline, which we know needs to look like a single job (in this new sense); that's where we came in. - A shell structure of code being executed together (details such as where we need new subjobs TBD) such as a function or a complex command typed at the command line or in a particular part of a pipeline --- this is roughly what's referred to as a "list" in the code. This would have arbitrarily complicated subjobs which would vary depending on what was being executed within the structure. There would be no external process associated with the top level job unless we explicitly forked to make this happen; putting the current shell job into the background should now be as natural as putting a chunk of shell code into a pipeline not as the last element. The status of the job is the status of the last command to execute (which may be a "return" if it's a function). - A pipeline, a set of jobs (in this new sense --- maybe we need a better name to separate this from job control) each of which was one of the above two types. (I don't think pipelines nest directly in pipelines, you need some intervening shell structure otherwise they degenerate to the same pipeline.) The status of the job is either the status of the last subjob in the pipeline (NO_PIPE_FAIL) or uses the PIPE_FAIL rules. If it worked to the extent of allowing the removal of things like list_pipe and STAT_SUPERJOB it would have served its purpose well. We might even be able to move over to this gradually. As long as we can still search all jobs, we could introduce the new structures with the existing logic. For example, the pipestatus code could for now search every process associated with the current job, but would gradually get rewritten to pick out subjobs at the appropriate level. You might hope that as this process went on it would become easier to ask the question "when has this job finished"? Possibly the most unpleasant bit of this is making the hierarchy of calls in exec.c agree with this new structural hierarchy. It might be messy enough to put the kibosh on the whole thing --- I don't understand execpline() and execpline2() and I think it's necessary to do so. Whether this is ever going to come about... pws -- Peter Stephenson <p.w.stephenson@ntlworld.com> Web page now at http://homepage.ntlworld.com/p.w.stephenson/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Shell job structure 2013-10-08 20:44 ` Shell job structure Peter Stephenson @ 2013-10-25 5:02 ` Bart Schaefer 2013-10-25 10:00 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2013-10-25 5:02 UTC (permalink / raw) To: zsh-workers Just to tie this off, as we now have this mostly addressed: On Oct 8, 9:44pm, Peter Stephenson wrote: } Subject: Shell job structure } } On Mon, 07 Oct 2013 07:40:49 -0700 } Bart Schaefer <schaefer@brasslantern.com> wrote: } > - because external jobs can exit and be reaped in arbitrary order, even } > in a pipeline, the job table is used to keep track of which position } > in the array to update More specifically: The "procs" linked list in a job table entry keeps track of the array positions. The complication is .. } > - jobs that run in the current shell don't have a complete job table entry ... and the "incomplete" part is that there *is* a job table entry, but the current shell task doesn't have an entry in the "procs" list. } > [cf. all the gyrations to suspend/background a loop] and aren't "reaped" } > in the same code path as external jobs, so the wrong array position (or } > none at all) may get updated Because zsh "forks to the left" to keep the tail of the pipeline in the current shell, the status of the current shell task has to be added to the end of $pipestatus (internally "pipestats"), which means an accurate count of the number of other processes in the pipeline is essential. } > - the incorrect update depends on whether the external job exited AND got } > reaped before the current-shell job has completed, because of the way } > reaping updates the job table, so correctness is unpredictable This has been fixed by moving the update of pipestats to the point where the job table entry is deleted, instead of trying to do it at the point where individual processes have their status updated. } > - complex commands "in the current shell" may have external subjobs that } > need a separate pipestatus (this applies only at end of a pipeline) I believe this has also been worked around by the above, but one thing we haven't tested yet is what value $pipestatus has *inside* a loop. } The job table would become a list of top-level jobs, while the job } structure would have pointers to nested jobs. As I mentioned elsewhere, the procs list and "other" pointer into the job table serve this function. It still might be done more cleanly, but at least we avoided a full rewrite for now. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Shell job structure 2013-10-25 5:02 ` Bart Schaefer @ 2013-10-25 10:00 ` Peter Stephenson 2013-10-25 15:13 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-25 10:00 UTC (permalink / raw) To: zsh-workers On Thu, 24 Oct 2013 22:02:14 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > } The job table would become a list of top-level jobs, while the job > } structure would have pointers to nested jobs. > > As I mentioned elsewhere, the procs list and "other" pointer into the > job table serve this function. > > It still might be done more cleanly, but at least we avoided a full > rewrite for now. There's a wider problem of what happens inside nested control structures, in particular functions. Consider what happens when you use "pipestatus" inside a function, and then try to apply it to a pipeline, that happens to run that function at the right hand end, after the whole pipeline has finished. Logically, at least, this requires some kind of hierarchical handling of jobs, not processes --- although simply saving and restoring the current job might be good enough. Does the current system take account of this kind of thing properly? If so, maybe things are better than I thought. pws ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Shell job structure 2013-10-25 10:00 ` Peter Stephenson @ 2013-10-25 15:13 ` Bart Schaefer 2013-10-25 15:21 ` Peter Stephenson 0 siblings, 1 reply; 9+ messages in thread From: Bart Schaefer @ 2013-10-25 15:13 UTC (permalink / raw) To: zsh-workers On Oct 25, 11:00am, Peter Stephenson wrote: } Subject: Re: Shell job structure } } On Thu, 24 Oct 2013 22:02:14 -0700 } Bart Schaefer <schaefer@brasslantern.com> wrote: } > } The job table would become a list of top-level jobs, while the job } > } structure would have pointers to nested jobs. } > } > As I mentioned elsewhere, the procs list and "other" pointer into the } > job table serve this function. } > } > It still might be done more cleanly, but at least we avoided a full } > rewrite for now. } } There's a wider problem of what happens inside nested control } structures, in particular functions. Consider what happens when you use } "pipestatus" inside a function, and then try to apply it to a pipeline, } that happens to run that function at the right hand end, after the whole } pipeline has finished. You mean like this? % true | false | true | (){ false | true | false; print $pipestatus }; print $pipestatus 1 0 1 0 1 0 0 Here's a loop construct: % false | true | false | while true | false | true; do print $pipestatus; break; done; print $pipestatus 0 1 0 1 0 1 0 If that's not what you mean, then I'm not sure what is expected here. } Logically, at least, this requires some kind of } hierarchical handling of jobs, not processes --- although simply saving } and restoring the current job might be good enough. Does the current } system take account of this kind of thing properly? After 31879+31885, pipestats isn't assigned until the Job object is deleted (which only happens after all parts of the job are finished) *except* in one case: the job was suspended. I'm pretty sure there is still a race condition in that circumstance wherein the parent shell may decide the current-shell task is done because a component process hasn't yet received the signal. Anyway, the point is that if the outer pipeline doesn't assign pipestats until the current-shell construct at the tail has completed, then it's theoretically safe to use $pipstatus for pipelines inside that construct. } If so, maybe things are better than I thought. One thing I still don't understand about this is the exactly when the STAT_SUPERJOB flag is applied and whether that's going to affect some case that we haven't yet thought to test. There's a huge comment in exec.c that attempts to explain it, but ... There's a complication in that exec.c also says -- /* If this job has finished, we leave it as a * normal (non-super-) job. */ -- which I'm afraid might complicate the handling of pipestats in a way we're not expecting. This is all probably related to the race condition that I just mentioned, and indeed the big comment ends with: * The code for all this is distributed over three files (exec.c, jobs.c, * and signals.c) and none of them is a simple one. So, all in all, there * may still be bugs, but considering the complexity (with race conditions, * signal handling, and all that), this should probably be expected. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Shell job structure 2013-10-25 15:13 ` Bart Schaefer @ 2013-10-25 15:21 ` Peter Stephenson 2013-10-25 17:04 ` Bart Schaefer 0 siblings, 1 reply; 9+ messages in thread From: Peter Stephenson @ 2013-10-25 15:21 UTC (permalink / raw) To: zsh-workers On Fri, 25 Oct 2013 08:13:11 -0700 Bart Schaefer <schaefer@brasslantern.com> wrote: > You mean like this? > > % true | false | true | (){ false | true | false; print $pipestatus }; print $pipestatus > 1 0 1 > 0 1 0 0 > > Here's a loop construct: > > % false | true | false | while true | false | true; do print $pipestatus; break; done; print $pipestatus > 0 1 0 > 1 0 1 0 That's the sort of thing I'm worrying about. As long as it can identify succesfully which job it needs to worry about at any time, it should be OK. pws ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Shell job structure 2013-10-25 15:21 ` Peter Stephenson @ 2013-10-25 17:04 ` Bart Schaefer 0 siblings, 0 replies; 9+ messages in thread From: Bart Schaefer @ 2013-10-25 17:04 UTC (permalink / raw) To: zsh-workers On Oct 25, 4:21pm, Peter Stephenson wrote: } } > % false | true | false | while true | false | true; do print $pipestatus; break; done; print $pipestatus } > 0 1 0 } > 1 0 1 0 } } That's the sort of thing I'm worrying about. As long as it can identify } succesfully which job it needs to worry about at any time, it should be } OK. Every entry in $pipestatus except the very last one is copied from a "Process" structure in the Job "procs" list, so as long as child reaping updates the correct Process (which seems to be pretty solid or we'd have scads of other problems) there's only that last slot to worry about. The global "lastval" is used to populate that final slot when the command is executing in the current shell. That's a bit fragile and somewhat backward because in all other cases the status that goes into pipestats is also used to *assign* lastval -- it's just this "optimized" case where we assume lastval has already been set correctly. This is why update_job() has to skip PIPEFAIL handling and leave that to printjob(), so they don't use lastval in the wrong order. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2013-10-25 17:05 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CADv1Z=pM7H4Xg9+GyWd4zw0cv0mXfbJvqip6vc7e_yrXRN=1sg@mail.gmail.com> [not found] ` <20131005223159.25fea6a0@pws-pc.ntlworld.com> 2013-10-07 0:36 ` No pipefail option? Bart Schaefer 2013-10-07 9:25 ` Peter Stephenson 2013-10-07 14:40 ` Bart Schaefer 2013-10-08 20:44 ` Shell job structure Peter Stephenson 2013-10-25 5:02 ` Bart Schaefer 2013-10-25 10:00 ` Peter Stephenson 2013-10-25 15:13 ` Bart Schaefer 2013-10-25 15:21 ` Peter Stephenson 2013-10-25 17:04 ` Bart Schaefer
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).