Re: No pipefail option?

Re: No pipefail option?

[Bart Schaefer]

[>workers]

On Oct 5, 10:31pm, Peter Stephenson wrote:
}
} Indeed, if $pipestatus is working as advertised, it really ought to be
} this simple...

Unfortunately, $pipestatus is not working as advertised.

See this thread from 2011:

http://www.zsh.org/mla/workers/2011/msg01394.html

In particular

http://www.zsh.org/mla/workers/2011/msg01396.html
http://www.zsh.org/mla/workers/2011/msg01470.html
http://www.zsh.org/mla/workers/2011/msg01475.html
http://www.zsh.org/mla/workers/2011/msg01476.html

The patch mentined in msg01476 is here:

http://www.zsh.org/mla/workers//2011/msg01472.html

But it didn't resolve the issue, so I never committed it.

The long and short of it is that pipestatus only works reliably when
all the commands in the pipeline are external programs.

Re: No pipefail option?

[Peter Stephenson]

On Sun, 06 Oct 2013 17:36:21 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> [>workers]
> 
> On Oct 5, 10:31pm, Peter Stephenson wrote:
> }
> } Indeed, if $pipestatus is working as advertised, it really ought to be
> } this simple...
> 
> Unfortunately, $pipestatus is not working as advertised.

Looks that this might not be fatal for the most typical uses of
PIPEFAIL, since I suspect if you've got a loop at the end of the pipe
processing the results in the main shell you'd typically detect errors
at that point.

> See this thread from 2011:
> 
> http://www.zsh.org/mla/workers/2011/msg01394.html
> 
> In particular
> 
> http://www.zsh.org/mla/workers/2011/msg01396.html
> http://www.zsh.org/mla/workers/2011/msg01470.html
> http://www.zsh.org/mla/workers/2011/msg01475.html
> http://www.zsh.org/mla/workers/2011/msg01476.html
> 
> The patch mentined in msg01476 is here:
> 
> http://www.zsh.org/mla/workers//2011/msg01472.html
> 
> But it didn't resolve the issue, so I never committed it.
> 
> The long and short of it is that pipestatus only works reliably when
> all the commands in the pipeline are external programs.

It's not clear to me if (i) the problem is only relevant to the final
stage in the pipeline (as we fork for others anyway) (ii) you need a
complex command rather than a simple builtin to get confused, but
presumably a complex command might be a function as there's nothing
special about jobs run from inside a function (they are not encapsulated
as far as job control is concerned).

I haven't yet quite worked out why it's not possible to detect the point
at which the job is finished and examine the state at that point, so I
can't comment further, but I might look again and ask stupid questions.

pws

Re: No pipefail option?

[Bart Schaefer]

On Oct 7, 10:25am, Peter Stephenson wrote:
}
} I haven't yet quite worked out why it's not possible to detect the point
} at which the job is finished and examine the state at that point, so I
} can't comment further, but I might look again and ask stupid questions.

I don't remember clearly all the details, but it's some combination of:

- because external jobs can exit and be reaped in arbitrary order, even
  in a pipeline, the job table is used to keep track of which position
  in the array to update

- jobs that run in the current shell don't have a complete job table entry
  [cf. all the gyrations to suspend/background a loop] and aren't "reaped"
  in the same code path as external jobs, so the wrong array position (or
  none at all) may get updated

- the incorrect update depends on whether the external job exited AND got
  reaped before the current-shell job has completed, because of the way
  reaping updates the job table, so correctness is unpredictable

- complex commands "in the current shell" may have external subjobs that
  need a separate pipestatus (this applies only at end of a pipeline)

Shell job structure

[Peter Stephenson]

On Mon, 07 Oct 2013 07:40:49 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> - because external jobs can exit and be reaped in arbitrary order, even
>   in a pipeline, the job table is used to keep track of which position
>   in the array to update
> 
> - jobs that run in the current shell don't have a complete job table entry
>   [cf. all the gyrations to suspend/background a loop] and aren't "reaped"
>   in the same code path as external jobs, so the wrong array position (or
>   none at all) may get updated
> 
> - the incorrect update depends on whether the external job exited AND got
>   reaped before the current-shell job has completed, because of the way
>   reaping updates the job table, so correctness is unpredictable
> 
> - complex commands "in the current shell" may have external subjobs that
>   need a separate pipestatus (this applies only at end of a pipeline)

This may be pie in the sky, but probably what we need is for jobs to be
hierarchical.  A job at the top level would always be treated as
separate from another job at the top level for the original purpose of
job control but might include nested jobs, representing objects such as
functions and chunks of pipelines, that might themselves have jobs
representing external commands.  Furthermore, a job would become a first
class representation of shell state, so anything just involving shell
builtins would have a job, removing the obfuscation in the current
relationship between jobs and code execution.  It would thus in
principle allow better encapsulation of other shell state.

The job table would become a list of top-level jobs, while the job
structure would have pointers to nested jobs.  We might get rid of the
table all together and simply have a single pointer that got searched
for top-level jobs the same way those jobs got searched for subjobs.  If
this was inefficient for searching we could hash PIDs and the like; if
it was inefficient for memory management (but I don't see why it should
be, particularly, compared with memory management for anything else) we
could still have a pool of job structures.

I think this means we'd have a status associated with a job, not just a
process.  How you got the job status would depend on the nature of the
job.

I think a job would be one of:

- An external command with one main process and possibly auxiliary
processes; the status of the job is just that of the process.  This is
sort of a degenerate case of a pipeline but I'm wondering it might be
neater to make it just one part of a pipeline so auxiliary processes get
tied to the right main process.  This also makes an external command in
a pipeline more similar to the case of a shell construct in a pipeline,
which we know needs to look like a single job (in this new sense);
that's where we came in.

- A shell structure of code being executed together (details such as
where we need new subjobs TBD) such as a function or a complex command
typed at the command line or in a particular part of a pipeline --- this
is roughly what's referred to as a "list" in the code.  This would have
arbitrarily complicated subjobs which would vary depending on what was
being executed within the structure.  There would be no external process
associated with the top level job unless we explicitly forked to make
this happen; putting the current shell job into the background should
now be as natural as putting a chunk of shell code into a pipeline not
as the last element.  The status of the job is the status of the last
command to execute (which may be a "return" if it's a function).

- A pipeline, a set of jobs (in this new sense --- maybe we need a
better name to separate this from job control) each of which was one of
the above two types.  (I don't think pipelines nest directly in
pipelines, you need some intervening shell structure otherwise they
degenerate to the same pipeline.)  The status of the job is either the
status of the last subjob in the pipeline (NO_PIPE_FAIL) or uses the
PIPE_FAIL rules.

If it worked to the extent of allowing the removal of things like
list_pipe and STAT_SUPERJOB it would have served its purpose well.

We might even be able to move over to this gradually.  As long as we can
still search all jobs, we could introduce the new structures with the
existing logic.  For example, the pipestatus code could for now search
every process associated with the current job, but would gradually get
rewritten to pick out subjobs at the appropriate level.  You might hope
that as this process went on it would become easier to ask the question
"when has this job finished"?

Possibly the most unpleasant bit of this is making the hierarchy of
calls in exec.c agree with this new structural hierarchy.  It might be
messy enough to put the kibosh on the whole thing --- I don't understand
execpline() and execpline2() and I think it's necessary to do so.

Whether this is ever going to come about...

pws

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Shell job structure

[Bart Schaefer]

Just to tie this off, as we now have this mostly addressed:

On Oct 8,  9:44pm, Peter Stephenson wrote:
} Subject: Shell job structure
}
} On Mon, 07 Oct 2013 07:40:49 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > - because external jobs can exit and be reaped in arbitrary order, even
} >   in a pipeline, the job table is used to keep track of which position
} >   in the array to update

More specifically:  The "procs" linked list in a job table entry keeps
track of the array positions.  The complication is ..

} > - jobs that run in the current shell don't have a complete job table entry

... and the "incomplete" part is that there *is* a job table entry, but
the current shell task doesn't have an entry in the "procs" list.

} >   [cf. all the gyrations to suspend/background a loop] and aren't "reaped"
} >   in the same code path as external jobs, so the wrong array position (or
} >   none at all) may get updated

Because zsh "forks to the left" to keep the tail of the pipeline in the
current shell, the status of the current shell task has to be added to
the end of $pipestatus (internally "pipestats"), which means an accurate
count of the number of other processes in the pipeline is essential.

} > - the incorrect update depends on whether the external job exited AND got
} >   reaped before the current-shell job has completed, because of the way
} >   reaping updates the job table, so correctness is unpredictable

This has been fixed by moving the update of pipestats to the point where
the job table entry is deleted, instead of trying to do it at the point
where individual processes have their status updated.

} > - complex commands "in the current shell" may have external subjobs that
} >   need a separate pipestatus (this applies only at end of a pipeline)

I believe this has also been worked around by the above, but one thing
we haven't tested yet is what value $pipestatus has *inside* a loop.

} The job table would become a list of top-level jobs, while the job
} structure would have pointers to nested jobs.

As I mentioned elsewhere, the procs list and "other" pointer into the
job table serve this function.

It still might be done more cleanly, but at least we avoided a full
rewrite for now.

Re: Shell job structure

[Peter Stephenson]

On Thu, 24 Oct 2013 22:02:14 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> } The job table would become a list of top-level jobs, while the job
> } structure would have pointers to nested jobs.
> 
> As I mentioned elsewhere, the procs list and "other" pointer into the
> job table serve this function.
> 
> It still might be done more cleanly, but at least we avoided a full
> rewrite for now.

There's a wider problem of what happens inside nested control
structures, in particular functions.  Consider what happens when you use
"pipestatus" inside a function, and then try to apply it to a pipeline,
that happens to run that function at the right hand end, after the whole
pipeline has finished.  Logically, at least, this requires some kind of
hierarchical handling of jobs, not processes --- although simply saving
and restoring the current job might be good enough.  Does the current
system take account of this kind of thing properly?  If so, maybe things
are better than I thought.

pws

Re: Shell job structure

[Bart Schaefer]

On Oct 25, 11:00am, Peter Stephenson wrote:
} Subject: Re: Shell job structure
}
} On Thu, 24 Oct 2013 22:02:14 -0700
} Bart Schaefer <schaefer@brasslantern.com> wrote:
} > } The job table would become a list of top-level jobs, while the job
} > } structure would have pointers to nested jobs.
} > 
} > As I mentioned elsewhere, the procs list and "other" pointer into the
} > job table serve this function.
} > 
} > It still might be done more cleanly, but at least we avoided a full
} > rewrite for now.
} 
} There's a wider problem of what happens inside nested control
} structures, in particular functions.  Consider what happens when you use
} "pipestatus" inside a function, and then try to apply it to a pipeline,
} that happens to run that function at the right hand end, after the whole
} pipeline has finished.

You mean like this?

% true | false | true | (){ false | true | false; print $pipestatus }; print $pipestatus
1 0 1
0 1 0 0

Here's a loop construct:

% false | true | false | while true | false | true; do print $pipestatus; break; done; print $pipestatus
0 1 0
1 0 1 0

If that's not what you mean, then I'm not sure what is expected here.

} Logically, at least, this requires some kind of
} hierarchical handling of jobs, not processes --- although simply saving
} and restoring the current job might be good enough.  Does the current
} system take account of this kind of thing properly?

After 31879+31885, pipestats isn't assigned until the Job object is
deleted (which only happens after all parts of the job are finished)
*except* in one case: the job was suspended.  I'm pretty sure there is
still a race condition in that circumstance wherein the parent shell
may decide the current-shell task is done because a component process
hasn't yet received the signal.

Anyway, the point is that if the outer pipeline doesn't assign pipestats
until the current-shell construct at the tail has completed, then it's
theoretically safe to use $pipstatus for pipelines inside that construct.

} If so, maybe things are better than I thought.

One thing I still don't understand about this is the exactly when the
STAT_SUPERJOB flag is applied and whether that's going to affect some
case that we haven't yet thought to test.  There's a huge comment in
exec.c that attempts to explain it, but ...

There's a complication in that exec.c also says --

                        /* If this job has finished, we leave it as a
                         * normal (non-super-) job. */

-- which I'm afraid might complicate the handling of pipestats in a way
we're not expecting.  This is all probably related to the race condition
that I just mentioned, and indeed the big comment ends with:

 * The code for all this is distributed over three files (exec.c, jobs.c,
 * and signals.c) and none of them is a simple one. So, all in all, there
 * may still be bugs, but considering the complexity (with race conditions,
 * signal handling, and all that), this should probably be expected.

Re: Shell job structure

[Peter Stephenson]

On Fri, 25 Oct 2013 08:13:11 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> You mean like this?
> 
> % true | false | true | (){ false | true | false; print $pipestatus }; print $pipestatus
> 1 0 1
> 0 1 0 0
> 
> Here's a loop construct:
> 
> % false | true | false | while true | false | true; do print $pipestatus; break; done; print $pipestatus
> 0 1 0
> 1 0 1 0

That's the sort of thing I'm worrying about.  As long as it can identify
succesfully which job it needs to worry about at any time, it should be
OK.

pws

Re: Shell job structure

[Bart Schaefer]

On Oct 25,  4:21pm, Peter Stephenson wrote:
}
} > % false | true | false | while true | false | true; do print $pipestatus; break; done; print $pipestatus
} > 0 1 0
} > 1 0 1 0
} 
} That's the sort of thing I'm worrying about.  As long as it can identify
} succesfully which job it needs to worry about at any time, it should be
} OK.

Every entry in $pipestatus except the very last one is copied from a
"Process" structure in the Job "procs" list, so as long as child reaping
updates the correct Process (which seems to be pretty solid or we'd have
scads of other problems) there's only that last slot to worry about.

The global "lastval" is used to populate that final slot when the command
is executing in the current shell.  That's a bit fragile and somewhat
backward because in all other cases the status that goes into pipestats
is also used to *assign* lastval -- it's just this "optimized" case where
we assume lastval has already been set correctly.

This is why update_job() has to skip PIPEFAIL handling and leave that to 
printjob(), so they don't use lastval in the wrong order.