Re: pws-22: killing the ZSH loops problem

Re: pws-22: killing the ZSH loops problem

[Sven Wischnowsky]

Bart Schaefer wrote:

> On Jun 18, 10:55am, Sven Wischnowsky wrote:
> } Subject: Re: pws-22: killing the ZSH loops problem
> }
> } Unless I'm missing something obvious, we can't simply execute
> } processes in a loop (or other shell construct) in the same pgrp as the 
> } shell. Sure, this would give us the SIGINT, but it would also give us
> } the SIGSTOP -- suspending the shell.
> 
> Strictly speaking, it would give us SIGTSTP -- ^Z does not generate STOP.
> Since TSTP is catchable (unlike STOP), the parent shell simply has to have
> a TSTP handler or ignore TSTP -- which zsh already does.

Oh, yes. Oops. I had tried implementing that and got a SIGSTOP --
without further thinking I wrote the above. Seems like the stuff I had 
hacked together caused update_job() to send a SIGSTOP to the parent
shell.
Hm. So there might be a way...

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

RE: pws-22: killing the ZSH loops problem

[Andrej Borsenkow]

> }
> } > } We need it only if MONITOR is set
> } >
> } > Not true!  MONITOR only affects handling of ^Z, not of ^C.  We need to be
> } > able to properly interrupt such loops in any shell.
> }
> } Ahem ... yes. What about "only in interactive shells?"
>
> What about "kill -INT"?  What about a shell script that isn't interactive
> but that's running in the foreground?
>

The shell script running in foreground will get INT from tty because it is in
TTY group and can do anything with it. The same, if I do kill -INT. The problem
only ever happens with loops started as part of shells with MIONITOR option set,
because it runs part of command (loop) in other process group as loop itself.
That prevents it from seeing this INT.

> } If don't miss something again: we have to fork only for the loop as whole.
>
> That's right, but the reason we don't do this now is so that parameter
> assignments inside the loop are visible after the loop terminates.  That
> doesn't work in most other shells.
>

As I understand, the last part of pipeline runs in current shell, and Sven's
suggestion was basically to run while ... done loop as <dummy command> | while
... loop. May be, I was wrong.

And another way is to run commands in the process group as shell and catch INT.
Is it even remotely possible?

> } ... so, it seems, execution time penalty is acceptable.  After all, you
> } don't use loops on every prompt.
>
> The execution time isn't as much at issue as is the process table space.
>

If it is needed (as I still believe) only for loops, started at PS1 when MONITOR
is set - it hardly happens too often. And having to kill your login shell to get
out of loop is not as nice

Re: pws-22: killing the ZSH loops problem

[Bart Schaefer]

On Jun 21, 11:08am, Andrej Borsenkow wrote:
} Subject: RE: pws-22: killing the ZSH loops problem
}
} > } We need it only if MONITOR is set
} >
} > Not true!  MONITOR only affects handling of ^Z, not of ^C.  We need to be
} > able to properly interrupt such loops in any shell.
} 
} Ahem ... yes. What about "only in interactive shells?"

What about "kill -INT"?  What about a shell script that isn't interactive
but that's running in the foreground?

} If don't miss something again: we have to fork only for the loop as whole.

That's right, but the reason we don't do this now is so that parameter
assignments inside the loop are visible after the loop terminates.  That
doesn't work in most other shells.

} ... so, it seems, execution time penalty is acceptable.  After all, you
} don't use loops on every prompt.

The execution time isn't as much at issue as is the process table space.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: pws-22: killing the ZSH loops problem

[Andrej Borsenkow]

> }
> } We need it only if MONITOR is set
>
> Not true!  MONITOR only affects handling of ^Z, not of ^C.  We need to be
> able to properly interrupt such loops in any shell.
>

Ahem ... yes. What about "only in interactive shells?"

> However, we only need it when (a) there's at least one external command
> (you can already interrupt purely builtin loops, try it) and (b) that
> external command doesn't return the proper exit status for zsh to learn
> that it caught a signal.  We can't know (b) in advance, so I can't come
> up with anything useful for (a) except to fork twice on every external
> command, which is horribly wasteful.
>

If don't miss something again: we have to fork only for the loop as whole. In
other words, only if the command being executed (top-level command) is a complex
command. This means, that in following

while xxx
  while yyy
  done
done

we only fork once for the outer loop and only, if it was input to PS1. It means,
we don't need to fork for the loops in functions (because theay are *not* zsh
commands), so, it seems, execution time penalty is acceptable. After all, you
don't use loops on every prompt.

> } And it just occured to me - is it possible to kill/stop/resume shell
> } function?
>
> Yes.  In fact, that was the recommended workaround for several pipe-into-
> a-loop problems before they gradually got fixed:

Well, thanks for reminder. Still, I'd really prefer properly running loops :-)

/andrej

Re: pws-22: killing the ZSH loops problem

[Bart Schaefer]

On Jun 18, 10:55am, Sven Wischnowsky wrote:
} Subject: Re: pws-22: killing the ZSH loops problem
}
} Unless I'm missing something obvious, we can't simply execute
} processes in a loop (or other shell construct) in the same pgrp as the 
} shell. Sure, this would give us the SIGINT, but it would also give us
} the SIGSTOP -- suspending the shell.

Strictly speaking, it would give us SIGTSTP -- ^Z does not generate STOP.
Since TSTP is catchable (unlike STOP), the parent shell simply has to have
a TSTP handler or ignore TSTP -- which zsh already does.

Incidentally, and probably not strictly a zsh thing, `kill -STOP $$` in an
xterm locks up the whole xterm (it won't even repaint itself or bring up
the ctrl-buttonN menus) and closing the xterm with the window manager
orphans the shell even if you've restarted it with `kill -CONT ...` from
somewhere else.  I had to `kill -9 ...` it.

On Jun 18,  7:33pm, Andrej Borsenkow wrote:
} Subject: RE: pws-22: killing the ZSH loops problem
}
} > (Btw. `foo | while ...' can be ^C'ed because we have the `foo' to find
} > that out. This means that we could make normal loops be ^C'able by
} > forking of a sub-shell for every loop and let the sub-shell do
} > nothing. Then ^C would SIGINT the sub-shell and the parent shell would
} > be notified about this -- but this is really ugly isn't it? Or should
} > we? But that would be an extra fork on every shell construct...)
} 
} We need it only if MONITOR is set

Not true!  MONITOR only affects handling of ^Z, not of ^C.  We need to be
able to properly interrupt such loops in any shell.

However, we only need it when (a) there's at least one external command
(you can already interrupt purely builtin loops, try it) and (b) that
external command doesn't return the proper exit status for zsh to learn
that it caught a signal.  We can't know (b) in advance, so I can't come
up with anything useful for (a) except to fork twice on every external
command, which is horribly wasteful.

} And it just occured to me - is it possible to kill/stop/resume shell
} function?

Yes.  In fact, that was the recommended workaround for several pipe-into-
a-loop problems before they gradually got fixed:  Put the loop in a shell
function and pipe to the function.  That's still the best way to minimize
the number of processes that get forked (there was a thread about this a
while back, perhaps as much as a year ago).

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

RE: pws-22: killing the ZSH loops problem

[Andrej Borsenkow]

>
> (Btw. `foo | while ...' can be ^C'ed because we have the `foo' to find
> that out. This means that we could make normal loops be ^C'able by
> forking of a sub-shell for every loop and let the sub-shell do
> nothing. Then ^C would SIGINT the sub-shell and the parent shell would
> be notified about this -- but this is really ugly isn't it? Or should
> we? But that would be an extra fork on every shell construct...)
>

We need it only if MONITOR is set and this is typically (if not exclusively)
interactive shells. Unfortunately, with new completion, we use loops heavily.
OTOH exactly with new completion in mind, there must be some way to kill it. It
is not good, when completion function hangs in some loop without any possibility
to get out. And it just occured to me - is it possible to kill/stop/resume shell
function?

> And another `btw.': comparison with bash shows that it has the same
> problem ^C'ing such commands in loops and behaves like zsh. But it
> can't even correctly stop such loops.
>
>

I looked at how it is implemented in ksh and csh on our system.

csh:

resets process group for every external command. It is impossible to ^C loop
(for the same reason as in Zsh). It is possible to ^Z it - but it makes loop
disapper. After fg only the last started external command is brought in
foreground. When it exits, loop effectively terminates. Looks horribly broken.

ksh:

does not reset process group at all. Ignores ^Z (TSTP). It is possible to ^C
loop (ksh cathes INT). It is impossible to ^Z loop because TSTP remains ignored
by child (I suspect actually a bug in ksh - it should reset TSTP to default
before exec'ing)

So - assuming, that we won't reset process group - is it possible to make zsh
catch INT (QUIT?) and TSTP and behave nicely?

/andrej

Re: pws-22: killing the ZSH loops problem

[Sven Wischnowsky]

I wrote:

> Bart Schaefer wrote:
> 
> > One possibility is to not permit job control of individual external jobs
> > run within a shell construct; that is, force ^Z to stop the entire shell
> > loop and restart it.  As has been mentioned before, this is easy in other
> > shells because they typically fork off the tails of pipelines whereas zsh
> > always forks off the heads -- but some of the new list_pipe code that was
> > added recently may give us the necessary hooks to manage it.  Given that,
> > we can stop using new pgrps for subjobs of a shell construct, and then
> > zsh can get the terminal signals directly again.
> 
> I think, this is the way to go. I'll have a look at it, but probably not
> before the weekend.

I was to hasty to agree with this...

Unless I'm missing something obvious, we can't simply execute
processes in a loop (or other shell construct) in the same pgrp as the 
shell. Sure, this would give us the SIGINT, but it would also give us
the SIGSTOP -- suspending the shell.

So the patch below at least makes loops suspendable as a whole, even
if they are not part of a pipeline (it was strange that we could do it 
for `foo | while ...' but not for `while ...'). For the ^C'ing of such 
loops I don't see a good solution any more -- other than this: with the
patch you can ^Z the loop, fg it and then you can ^C it if the next
external command has been started, because then the sub-shell created
when the loop was suspended gets the SIGINT, too, and terminates.

(Btw. `foo | while ...' can be ^C'ed because we have the `foo' to find 
that out. This means that we could make normal loops be ^C'able by
forking of a sub-shell for every loop and let the sub-shell do
nothing. Then ^C would SIGINT the sub-shell and the parent shell would 
be notified about this -- but this is really ugly isn't it? Or should
we? But that would be an extra fork on every shell construct...)

And another `btw.': comparison with bash shows that it has the same
problem ^C'ing such commands in loops and behaves like zsh. But it
can't even correctly stop such loops.


Bye
 Sven

diff -u oos/exec.c Src/exec.c
--- oos/exec.c	Fri Jun 18 08:50:07 1999
+++ Src/exec.c	Fri Jun 18 08:50:23 1999
@@ -919,7 +919,7 @@
 	    }
 
 	    for (; !nowait;) {
-		if (list_pipe_child) {
+		if (list_pipe_child || pline_level) {
 		    jn->stat |= STAT_NOPRINT;
 		    makerunning(jn);
 		}
@@ -930,8 +930,8 @@
 		    jn->stat & STAT_DONE &&
 		    lastval2 & 0200)
 		    killpg(mypgrp, lastval2 & ~0200);
-		if ((list_pipe || last1) && !list_pipe_child &&
-		    jn->stat & STAT_STOPPED) {
+		if ((list_pipe || last1 || pline_level) &&
+		    !list_pipe_child && jn->stat & STAT_STOPPED) {
 		    pid_t pid;
 		    int synch[2];
 
@@ -961,7 +961,8 @@
 			jobtab[list_pipe_job].stat |= STAT_SUPERJOB;
 			jn->stat |= STAT_SUBJOB | STAT_NOPRINT;
 			jn->other = pid;
-			killpg(jobtab[list_pipe_job].gleader, SIGSTOP);
+			if (list_pipe || last1)
+			    killpg(jobtab[list_pipe_job].gleader, SIGSTOP);
 			break;
 		    }
 		    else {
@@ -988,7 +989,8 @@
 		jn->stat |= STAT_NOPRINT;
 		killjb(jobtab + pj, lastval & ~0200);
 	    }
-	    if (list_pipe_child || (list_pipe && (jn->stat & STAT_DONE)))
+	    if (list_pipe_child || ((list_pipe || pline_level) &&
+				    (jn->stat & STAT_DONE)))
 		deletejob(jn);
 	    thisjob = pj;
 

--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: pws-22: killing the ZSH loops problem

[Sven Wischnowsky]

Bart Schaefer wrote:

> One possibility is to not permit job control of individual external jobs
> run within a shell construct; that is, force ^Z to stop the entire shell
> loop and restart it.  As has been mentioned before, this is easy in other
> shells because they typically fork off the tails of pipelines whereas zsh
> always forks off the heads -- but some of the new list_pipe code that was
> added recently may give us the necessary hooks to manage it.  Given that,
> we can stop using new pgrps for subjobs of a shell construct, and then
> zsh can get the terminal signals directly again.

I think, this is the way to go. I'll have a look at it, but probably not
before the weekend.

Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: pws-22: killing the ZSH loops problem

[Bart Schaefer]

On Jun 16,  5:14pm, Peter Stephenson wrote:
} Subject: Re: pws-22: killing the ZSH loops problem
}
} "Andrej Borsenkow" wrote:
} > bor@itsrm2:~%> while true
} > while> do
} > while> echo $((i++))
} > while> sh -c "read line"
} > while> done
} > 0
} > ^C
} > 1
} > ^C
} > 2
} 
} So there are three cases (1) no job control, like sh, (2) job control and
} the sh in the while loop goes into a different pgrp, (3) job control but
} the sh in the while loop stays in the pgrp of the controlling shell.  Cases
} (1) and (3) exit cleanly, because the shell itself gets the ^C; case (2),
} which is what zsh is doing, doesn't because it doesn't.

Yes, I believe this is correct.  It's related to this comment in jobs.c:
    /* When MONITOR is set, the foreground process runs in a different *
     * process group from the shell, so the shell will not receive     *
     * terminal signals, therefore we we pretend that the shell got    *
     * the signal too.                                                 */

If the foreground job catches INT and doesn't exit with the proper status,
zsh can't tell anything other than that it exited -- just as Andrej found
by experiment.

} So maybe zsh should not set a new process group for foreground processes
} run within shell structures.  Sven, any ideas about doing this?  It might
} clash with job control for such processes, which are currently handled by
} forking the shell and setting its pgrp to that of the other jobs; this
} wouldn't be possible any more, so we're going to have to live with
} something not working.

One possibility is to not permit job control of individual external jobs
run within a shell construct; that is, force ^Z to stop the entire shell
loop and restart it.  As has been mentioned before, this is easy in other
shells because they typically fork off the tails of pipelines whereas zsh
always forks off the heads -- but some of the new list_pipe code that was
added recently may give us the necessary hooks to manage it.  Given that,
we can stop using new pgrps for subjobs of a shell construct, and then
zsh can get the terminal signals directly again.

Unfortunately this might require shoveling around some bits of context
information that aren't presently available, in order to prevent the
execution code from creating the new process group.  I haven't looked.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Re: pws-22: killing the ZSH loops problem

[Peter Stephenson]

"Andrej Borsenkow" wrote:
> bor@itsrm2:~%> while true
> while> do
> while> echo $((i++))
> while> sh -c "read line"
> while> done
> 0
> ^C
> 1
> ^C
> 2

I've formed a hypothesis based on different versions of ksh.

On some versions, the `sh -c' goes into a different process group: here, I
get the same problem as zsh.  On others, the process group of the sh is the
same as that of the ksh: then the loop exits cleanly.

So there are three cases (1) no job control, like sh, (2) job control and
the sh in the while loop goes into a different pgrp, (3) job control but
the sh in the while loop stays in the pgrp of the controlling shell.  Cases
(1) and (3) exit cleanly, because the shell itself gets the ^C; case (2),
which is what zsh is doing, doesn't because it doesn't.

So maybe zsh should not set a new process group for foreground processes
run within shell structures.  Sven, any ideas about doing this?  It might
clash with job control for such processes, which are currently handled by
forking the shell and setting its pgrp to that of the other jobs; this
wouldn't be possible any more, so we're going to have to live with
something not working.

I haven't quite understood why a zsh in the loop works properly (i.e. the
loop exits cleanly when interrupted), but it seems to be nothing to do with
process groups: the `zsh -c' is exiting with the correct status
(i.e. WIFSIGNALED(...) returns true so the parent shell know what
happened).  Looks like we got something right.  Arguably we can blame the
problem on all those shells which don't do this :-).  (Next we find it's
only because of a bug that zsh works this way...)

-- 
Peter Stephenson <pws@ibmth.df.unipi.it>       Tel: +39 050 844536
WWW:  http://www.ifh.de/~pws/
Dipartimento di Fisica, Via Buonarroti 2, 56127 Pisa, Italy

RE: pws-22: killing the ZSH loops problem

[Andrej Borsenkow]

>
> It does not happen, if I use 'zsh -c "read line"' (that is, the whole
> loop is killed), but as it stays with every external command I tried.
>


Sorry for all these followups. It looks, like the problem happens with programs,
that catch SIGINT (sh, gzip, gzcat) but not with programs, that leave it default
(cat, zsh). In the latter case, Zsh gets information about child being
interraupted by SIGINT and reacts correctly. In the former case, Zsh only knows,
that program (normally) exited for whatever reason.

Looks, like Zsh cannot rely on child info and must handle INT itself. I wonder,
how 3.0.5 behaves in this case.

/andrej

RE: pws-22: killing the ZSH loops problem

[Andrej Borsenkow]

 The easy way to reproduce it is
>
> bor@itsrm2:~%> while true
> while> do
> while> echo $((i++))
> while> sh -c "read line"
> while> done
> 0
> ^C
> 1
> ^C
> 2
>


It does not happen, if I use 'zsh -c "read line"' (that is, the whole loop is
killed), but as it stays with every external command I tried.

/andrej

pws-22: killing the ZSH loops problem

[Andrej Borsenkow]

I wanted to gzip a bunch of files, so I did

for i in *.txt
do
gzip $i
done

I overlooked that I had (plenty of) already gzipped versions, so after I hitted
a couple of times "n" (as a reply to gzip, if I want to overwrite it) the next
time I just hitted ^C to stop the whole story.  It did not. It just killed the
current gzip and for loop happily ran the next one. The easy way to reproduce it
is

bor@itsrm2:~%> while true
while> do
while> echo $((i++))
while> sh -c "read line"
while> done
0
^C
1
^C
2

Note, that hitting ^C simply kills started command instead of killing the whole
loop. The same happens with pws-19. ^Z does not help either - it stops the
current inner command.

Compare it with shell:

bor@itsrm2:~%> sh
$ i=0
$ while true
> do
> i=`expr "$i + 1"`
> echo $i
> sh -c "read line"
> done
0 + 1

0 + 1 + 1

0 + 1 + 1 + 1
^C
$

(sorry, expr was wrong, but I have not used it for quite a long time). Note,
that in shell it kills the whole loop.

/andrej