What's a superjob?

What's a superjob?

[Daniel Shahaf]

Could someone explain in a sentence what a 'superjob' is?

I can see (from 'struct job' and grepping around) that it's a job that
has subjobs.  I can see that STAT_SUPERJOB is set in execpline() when
something happens with list_pipe_job, but list_pipe_job is also
undocumented.

I'm guessing that it's the 'bodyguard shell' demonsrated in
Joey's 43506 and explained in 43514.  Is that correct?

Thanks

Daniel

Re: What's a superjob?

[Peter Stephenson]

On Wed, 26 Sep 2018 15:31:23 +0000
Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Could someone explain in a sentence what a 'superjob' is?

Not in a sentence, no.

> I'm guessing that it's the 'bodyguard shell' demonsrated in
> Joey's 43506 and explained in 43514.  Is that correct?

Yes, basically, if you mean what I think you mean.

% e() {
  vi
  echo "There might be some more code here, for the sake of example"
}
% e

Now we have just one process running: the main shell, and vi.  Suppose
you hit ^Z.  Then vi suspends.  The main shell is in the middle of
running e, but it's not supposed to finish doing that till vi exits.
But without special action we're now back at the top-level shell prompt
and it never will execute.

So when it detects this case the shell forks.  Now we have two processes
apart from the main shell itself: the forked shell is in the superjob, and
will eventually finish running the function.  The vi process is in the
subjob.  The superjob may have more than one process if the structure of
the code is more complicated e.g. a pipeline that needs to finish only
when the subjob is done.

Why two different jobs?  Because we have two processes separately forked
from the main shell.  When the subjob exits, the main shell will tell
the superjob to carry on, so it will print that echo --- that's why the
forked shell is the "super"job, not the other way around, and why we
show the superjob to the user rather than the subjob, because it's the
longer lived of the two.

We are having to jump through hoops because the user has been told there
is only the superjob.

On

% bg

both jobs are put into the background, though the forked shell
(superjob) will stay suspended for the time being.  In this case, the
subjob gets SIGTTOU and we now report *this* suspension (we suppress the
fact that the superjob's forked shell process is suspended as it's not
relevant to what the user thinks is going on).

On

% fg

both jobs are put back into the foreground, but again the forked shell
stays suspended.

When vi (subjob) exits, the forked shell is woken up and runs the rest
of the function.

pws

Re: What's a superjob?

[Bart Schaefer]

On Wed, Sep 26, 2018 at 8:51 AM Peter Stephenson
<p.stephenson@samsung.com> wrote:
>
> Now we have just one process running: the main shell, and vi.  Suppose
> you hit ^Z.  Then vi suspends.  The main shell is in the middle of
> running e, but it's not supposed to finish doing that till vi exits.
> But without special action we're now back at the top-level shell prompt
> and it never will execute.

To add a bit more color here ... in the absence of this special
handling, when vi is suspended, the shell has two choices:
1) treat vi as successfully completed and continue executing the function, or
2) treat vi as failed and stop the function at this point.

IIRC different shells have historically made different decisions about
this.  I believe zsh's original behavior was #1, but that led to
problems if (in this type of example) the remainder of the function
depends on the content of the edited file.  So #2 avoids side-effects,
but causes at minimum annoyance because now you have a suspended vi
but the rest of the function is simply gone.

The other possibility is to immediately fork the whole function as
soon as any non-builtin command is encountered, and then have the
forked function fork again for the external job, to keep the process
tree "normal".  This would simplify the job handling in the main
shell, but ruin the intention of the function being able to affect the
main shell context in the case where no job control ever occurs.

Re: What's a superjob?

[Bart Schaefer]

On Wed, Sep 26, 2018 at 10:42 AM Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> The other possibility is to immediately fork the whole function as
> soon as any non-builtin command is encountered, and then have the
> forked function fork again for the external job, to keep the process
> tree "normal".  This would simplify the job handling in the main
> shell, but ruin the intention of the function being able to affect the
> main shell context in the case where no job control ever occurs.

Hmm, now that Peter has already finished his juggling act, caught all
the clubs, and taken a bow, this suggestion may be moot, but:

Another possibility would be to always fork twice for any external job
inside a current shell construct, so we immediately end up with this:
  zsh[parent] -> zsh[superjob] -> vi[subjob]
* If subjob exits normally, so does the superjob; parent continues on
with the function/brace expression/loop/whatever.
* However, if subjob is suspended, superjob is also suspended and
changes its state such that when subjob finally exits, superjob
continues with whatever.
* In parent, if superjob is suspended, the rest of the construct is
abandoned (because we know superjob will finish it).

This wastes a fork if job control never happens, but it means the
parent only has to manage one job at a time.  The extra fork could be
skipped if job control is disabled.

[PATCH] Start documenting jobs.c, in particular superjobs.

[Daniel Shahaf]

---
 Src/jobs.c | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/Src/jobs.c b/Src/jobs.c
index 8103f5c92..c15001d6b 100644
--- a/Src/jobs.c
+++ b/Src/jobs.c
@@ -30,6 +30,27 @@
 #include "zsh.mdh"
 #include "jobs.pro"
 
+/*
+ * Job control in zsh
+ * ==================
+ *
+ * A 'job' represents a pipeline; see the section JOBS in zshmisc(1)) for an
+ * introduction.  The 'struct job's are allocated in the array 'jobtab' which
+ * has 'jobtabsize' elements.  The job whose processes we are currently
+ * preparing to execute is identified by the global variable 'thisjob'.
+ *
+ * A 'superjob' is a job that represents a complex shell construct that has been
+ * backgrounded.  For example, if one runs '() { vi; echo }', a job is created
+ * for the pipeline 'vi'.  If one then backgrounds vi (with ^Z / SIGTSTP), 
+ * the shell forks; the parent shell returns to the interactive prompt and
+ * the child shell becomes a new job in the parent shell.  The job representing
+ * the child shell to the parent shell is a superjob (STAT_SUPERJOB); the 'vi'
+ * job is marked as a subjob (STAT_SUBJOB) in the parent shell.  When the child
+ * shell is resumed (with fg / SIGCONT), it forwards the signal to vi and,
+ * after vi exits, continues executing the remainder of the function.
+ * (See workers/43565.)
+ */
+
 /* the process group of the shell at startup (equal to mypgprp, except
    when we started without being process group leader */
 
@@ -46,17 +67,17 @@ mod_export pid_t mypgrp;
 /**/
 pid_t last_attached_pgrp;
  
-/* the job we are working on */
+/* the job we are working on, or -1 if none */
  
 /**/
 mod_export int thisjob;
 
-/* the current job (+) */
+/* the current job (%+) */
  
 /**/
 mod_export int curjob;
  
-/* the previous job (-) */
+/* the previous job (%-) */
  
 /**/
 mod_export int prevjob;

Re: What's a superjob?

[Vincent Lefevre]

On 2018-09-26 10:42:00 -0700, Bart Schaefer wrote:
> On Wed, Sep 26, 2018 at 8:51 AM Peter Stephenson
> <p.stephenson@samsung.com> wrote:
> >
> > Now we have just one process running: the main shell, and vi.  Suppose
> > you hit ^Z.  Then vi suspends.  The main shell is in the middle of
> > running e, but it's not supposed to finish doing that till vi exits.
> > But without special action we're now back at the top-level shell prompt
> > and it never will execute.
> 
> To add a bit more color here ... in the absence of this special
> handling, when vi is suspended, the shell has two choices:
> 1) treat vi as successfully completed and continue executing the function, or
> 2) treat vi as failed and stop the function at this point.
> 
> IIRC different shells have historically made different decisions about
> this.  I believe zsh's original behavior was #1, but that led to
> problems if (in this type of example) the remainder of the function
> depends on the content of the edited file.  So #2 avoids side-effects,
> but causes at minimum annoyance because now you have a suspended vi
> but the rest of the function is simply gone.

But if the goal of the rest of the function is to modify the current
shell environment, this is just like if it were gone. Couldn't zsh
avoid the extra fork and immediately ask the user what to do?
For instance: continue in foreground (avoiding the extra fork),
do the extra fork, interrupt the function in some way.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)