Bug: ZSH crashes upon receiving SIGINT

Bug: ZSH crashes upon receiving SIGINT

[John]

Background:  Arch Linux[1] has a bash script used for installing the distro and or fixing a broken distro which is shipped with the live CD called 'arch-chroot.'
Bug:  ZSH is not capable of defending itself against SIGINT and crashes from within a chroot.  No other shell tested (
mksh, bash, dash, fish, and tcsh) seems to suffer from this deficiency.  In a recent bug report[2], the Arch devs believe this bug to be contained within ZSH, not within their code.

How to trigger the bug:
1) Mount an Arch partition and install 'zsh' within it
2) Execute the chroot command from another Arch env: arch-chroot /mnt/mini /bin/zsh
3) Hit ctrl+c in the chroot

Example of the bug:

# arch-chroot /mnt/mini /bin/zsh

#
<<hit ctrl-c>>
# umount: /mnt/mini/dev/pts: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)
umount: /mnt/mini/dev: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)
%
zsh: error on TTY read: Input/output error

Thank you for the time and I am glad to provide additional info or testing.
PLEASE CC ME ON ANY REPLY AS I AM NOT SUBSCRIBED TO THIS ML.

1. https://www.archlinux.org

2. https://bugs.archlinux.org/task/44073

Re: Bug: ZSH crashes upon receiving SIGINT

[Peter Stephenson]

On Mon, 9 Mar 2015 17:46:43 +0000 (UTC)
John <da_audiophile@yahoo.com> wrote:
> Background:  Arch Linux[1] has a bash script used for installing the
> distro and or fixing a broken distro which is shipped with the live CD
> called 'arch-chroot.' 
> Bug:  ZSH is not capable of defending itself against SIGINT and
> crashes from within a chroot.

From the report below it's obviously something more specific than this
that's happening.

> How to trigger the bug:
> 1) Mount an Arch partition and install 'zsh' within it
> 2) Execute the chroot command from another Arch env: arch-chroot
> /mnt/mini /bin/zsh
> 3) Hit ctrl+c in the chroot

I'm afraid the fact this description appears to be specific to that
distro and it looks like a lot of setting up of the chroot partition is
presupposed, so it's unlikely anyone on this list who can actually help
debug this is going to be able to reproduce it in this form.
Nonetheless, it does look like we have enough information at least to
start making suggestions, so if you're able to do a bit of further
prodding it's not hopeless...

> # arch-chroot /mnt/mini /bin/zsh
> 
> #
> <<hit ctrl-c>>
> # umount: /mnt/mini/dev/pts: target is busy
> (In some cases useful info about processes that
> use the device is found by lsof(8) or fuser(1).)
> umount: /mnt/mini/dev: target is busy
> (In some cases useful info about processes that
> use the device is found by lsof(8) or fuser(1).)
> %
> zsh: error on TTY read: Input/output error

Is that ^C happening at the command line, i.e. when no other
programme than the shell is running?  That's how I read the above, but
it would be good to be explicit.

In that case, what does the umount stuff mean?  That's certainly got
nothing to do with the shell's internal response to a Ctrl-C.

The error at the end comes from a specific point in the shell line
editor: if it finds it can't read from the TTY, and the error isn't one
of the small number it thinks are recoverable, it will exit.  In this
case the error appears to be EIO.  So this certainly ties in with the
behaviour you're seeing.

The question is where this error is coming from and it looks pretty
likely it's somehow associated with that "umount /mnt/mini/dev/pts" as
that looks like an attempt to remove the devices providing the ttys (I'm
guessing as is this is system stuff I don't know about).  The unmount
may have failed but may also have left the tty in a bad state.  I think
we need to know what's going on here.

To get anywhere within the shell, we need to know a little bit more
about the nature of the error and what we can do about it.  Clearly if
the terminal *has* become unusable, there's nothing we can do.  If it's
not happening with other shells, do you still see the errors to do with
"umount"?   If not, I'm afraid I'm stuck --- I just don't see any way
the shell can create system administration commands out of thin air.  If
they do, do you see any evidence of errors owing to bad terminal state?
Is there some shell framework installed by user commands, for example a
trap, that might be related to the unmounts?

My guess would be EIO isn't actually recoverable, so retrying wouldn't
help, though it would be fairly easy to add some test code to see.
Here's a useful summary from http://aplawrence.com/Unixart/errors.html

  #define EIO 5 /* I/O error */

  The catchall for all manner of unexpected hardware errors. It could be
  from a physical error, but additionally, an orphaned process (a
  process whose parent has died) that attempts to read from standard
  input will get this. BSD systems return this if you try to open a pty
  device that is already in use. An attempt to read from a stream that
  is closed will return EIO, as will a disk read or write that is
  outside of the physical bounds of the device. An open of /dev/tty when
  the process has no controlling tty will spit back EIO also.

It's possible zsh is somehow exacerbating the situation by performing
some operation like reopening the terminal, although it could be
anything --- I haven't looked for this and it's pure theorising but
something like this might explain why other shells soldier on but zsh
has problems.  The effect of this in the case we appear to be seeing
would be very system specific.  We can add test code for this, too, but
at the moment I think that's premature --- the evidence above suggests
something outside the shell is making life difficult and we need to
track that down first.

pws

Re: Bug: ZSH crashes upon receiving SIGINT

[Peter Stephenson]

On Tue, 10 Mar 2015 10:46:53 +0000
Peter Stephenson <p.w.stephenson@ntlworld.com> wrote:
> The error at the end comes from a specific point in the shell line
> editor: if it finds it can't read from the TTY, and the error isn't one
> of the small number it thinks are recoverable, it will exit.  In this
> case the error appears to be EIO.  So this certainly ties in with the
> behaviour you're seeing.

Minor update here: on looking a little more closely, I see the shell
will only exit on the second occurrence of EIO when reading from the
TTY.  On the first it will attempt to fix things up by forcing itself to
be the TTY foreground process.  It's possible this is part of the
puzzle.

pws

Re: Bug: ZSH crashes upon receiving SIGINT

[John]

----- Original Message -----
> 
> From the report below it's obviously something more specific than this
> that's happening.
> 
>>  How to trigger the bug:
>>  1) Mount an Arch partition and install 'zsh' within it
>>  2) Execute the chroot command from another Arch env: arch-chroot
>>  /mnt/mini /bin/zsh
>>  3) Hit ctrl+c in the chroot
> 
> I'm afraid the fact this description appears to be specific to that
> distro and it looks like a lot of setting up of the chroot partition is
> presupposed, so it's unlikely anyone on this list who can actually help
> debug this is going to be able to reproduce it in this form.
> Nonetheless, it does look like we have enough information at least to
> start making suggestions, so if you're able to do a bit of further
> prodding it's not hopeless...
> 

Firstly, thank you for your thoughtful reply.

Yes, I didn't think I would be fortunate enough to have one of you guys running Arch Linux.  If you have a spare partition free, you can just boot from the live CD (www.archlinux.org) and execute a few commands to have an Arch partition only for the purpose of seeing this bug/troubleshooting.  If this is acceptable to you, I am glad to distill down exactly what you would type to do this.  Please let me know.

>>  # arch-chroot /mnt/mini /bin/zsh
>> 
>>  #
>>  <<hit ctrl-c>>
>>  # umount: /mnt/mini/dev/pts: target is busy
>>  (In some cases useful info about processes that
>>  use the device is found by lsof(8) or fuser(1).)
>>  umount: /mnt/mini/dev: target is busy
>>  (In some cases useful info about processes that
>>  use the device is found by lsof(8) or fuser(1).)
>>  %
>>  zsh: error on TTY read: Input/output error
> 
> Is that ^C happening at the command line, i.e. when no other
> programme than the shell is running?  That's how I read the above, but
> it would be good to be explicit.

Yes, that is correct.  Upon running the first line (arch-chroot ...) the chroot binary is called and the next prompt is INSIDE the partition target (/mnt/mini in my example).  You can run commands inside the chroot just fine, but it breaks if you hit ctrl+c as I have denoted with the <<hit ctrl-c>> which probably wasn't very obvious without this explanation.

> In that case, what does the umount stuff mean?  That's certainly got
> nothing to do with the shell's internal response to a Ctrl-C.

The umount and subsequent lines are all thrown when you hit ctrl+c.

> The error at the end comes from a specific point in the shell line
> editor: if it finds it can't read from the TTY, and the error isn't one
> of the small number it thinks are recoverable, it will exit.  In this
> case the error appears to be EIO.  So this certainly ties in with the
> behaviour you're seeing.
> 
> The question is where this error is coming from and it looks pretty
> likely it's somehow associated with that "umount 
> /mnt/mini/dev/pts" as
> that looks like an attempt to remove the devices providing the ttys (I'm
> guessing as is this is system stuff I don't know about).  The unmount
> may have failed but may also have left the tty in a bad state.  I think
> we need to know what's going on here.

I agree with you.  Most of the arch-chroot script is over my head.  I have linked it here if the code within is meaningful and if you have the time to read through it.  It is only 309 lines: http://pastebin.com/yDyuqxqi

<snip>

I'm afraid the comments from Dave in the Arch Linux bug report (https://bugs.archlinux.org/task/44073) are all I have to go on as to root cause.  I am happy to try to troubleshoot if you do not want to try to make a simple Arch partition on your system and try yourself from the live CD.  Please let me know and thanks again.

Re: Bug: ZSH crashes upon receiving SIGINT

[Bart Schaefer]

On Mar 10, 10:06pm, John wrote:
}
} > we need to know what's going on here.
} 
} I agree with you. Most of the arch-chroot script is over my
} head. I have linked it here if the code within is meaningful and
} if you have the time to read through it. It is only 309 lines:
} http://pastebin.com/yDyuqxqi

I took at look at this script.  The "unmount" is coming from the
exit trap of the bash process that starts the whole thing off.

So I *think* what's happening is:

Bash starts "unshare" which becomes the TTY process group leader.

Unshare starts zsh which does not make itself the process group leader
(see previous -workers list discussions of zsh executing inside PID
namespaces, of which "unshare" creates one).

The ^C therefore is delivered to "unshare" which exits, ending the
bash script and executing the trap, which removes the tty device.

This causes zsh to get I/O error and exit.

It's *possible* that this is already "fixed" in development versions
of zsh, though most of the PID namespace changes had to do with avoiding
being process group leader rather than forcibily becoming so.  It's also
not clear that this is actually zsh's problem; it seems to me that the
arch-chroot script and/or "unshare" are not doing things they should be.

Re: Bug: ZSH crashes upon receiving SIGINT

[Peter Stephenson]

On Tue, 10 Mar 2015 17:43:27 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> So I *think* what's happening is:
> 
> Bash starts "unshare" which becomes the TTY process group leader.
> 
> Unshare starts zsh which does not make itself the process group leader
> (see previous -workers list discussions of zsh executing inside PID
> namespaces, of which "unshare" creates one).
> 
> The ^C therefore is delivered to "unshare" which exits, ending the
> bash script and executing the trap, which removes the tty device.
> 
> This causes zsh to get I/O error and exit.
> 
> It's *possible* that this is already "fixed" in development versions
> of zsh, though most of the PID namespace changes had to do with avoiding
> being process group leader rather than forcibily becoming so.  It's also
> not clear that this is actually zsh's problem; it seems to me that the
> arch-chroot script and/or "unshare" are not doing things they should be.

The PID namespace issue I remember is from the docker thread,
zsh-workers/34503.

http://www.zsh.org/mla/workers/2015/msg00427.html

I only remember the problem with unreaped children in that case, but
I'll refresh my mind when I get home.

pws

Re: Bug: ZSH crashes upon receiving SIGINT

[John]

> I took at look at this script.  The "unmount" is coming from the
> exit trap of the bash process that starts the whole thing off.
> 
> So I *think* what's happening is:
> 
> Bash starts "unshare" which becomes the TTY process group leader.
> 
> Unshare starts zsh which does not make itself the process group leader
> (see previous -workers list discussions of zsh executing inside PID
> namespaces, of which "unshare" creates one).
> 
> The ^C therefore is delivered to "unshare" which exits, ending the
> bash script and executing the trap, which removes the tty device.
> 
> This causes zsh to get I/O error and exit.
> 
> It's *possible* that this is already "fixed" in development 
> versions
> of zsh, though most of the PID namespace changes had to do with avoiding
> being process group leader rather than forcibily becoming so.  It's also
> not clear that this is actually zsh's problem; it seems to me that the
> arch-chroot script and/or "unshare" are not doing things they should 
> be.


Thank you for the reply.  Not too sure that I can add anything intelligent to the discussion, but I can confirm that the latest zsh from git (5.0.7.336.ge85906e) does not fix this issue.

Re: Bug: ZSH crashes upon receiving SIGINT

[Peter Stephenson]

On Wed, 11 Mar 2015 09:44:47 +0000
Peter Stephenson <p.stephenson@samsung.com> wrote:
> On Tue, 10 Mar 2015 17:43:27 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > So I *think* what's happening is:
> > 
> > Bash starts "unshare" which becomes the TTY process group leader.
> > 
> > Unshare starts zsh which does not make itself the process group leader
> > (see previous -workers list discussions of zsh executing inside PID
> > namespaces, of which "unshare" creates one).
> > 
> > The ^C therefore is delivered to "unshare" which exits, ending the
> > bash script and executing the trap, which removes the tty device.
> > 
> > This causes zsh to get I/O error and exit.
> > 
> > It's *possible* that this is already "fixed" in development versions
> > of zsh, though most of the PID namespace changes had to do with avoiding
> > being process group leader rather than forcibily becoming so.  It's also
> > not clear that this is actually zsh's problem; it seems to me that the
> > arch-chroot script and/or "unshare" are not doing things they should be.
> 
> The PID namespace issue I remember is from the docker thread,
> zsh-workers/34503.

I'm wrong, it's the discussion around 33992:

http://www.zsh.org/mla/workers/2014/msg01779.html

This does sound much more similar.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: Bug: ZSH crashes upon receiving SIGINT

[Peter Stephenson]

As an easy check of the shell setup, could you run

zmodload zsh/parameter
print $options[MONITOR]

in the problem case to see if job control has initialised properly?

pws

Re: Bug: ZSH crashes upon receiving SIGINT

[John]

It seems OK.

% zmodload zsh/parameter

print $options[MONITOR]
on


% sudo arch-chroot /mnt/mini /bin/zsh# << this is in the chroot >>
# << here I now hit ctrl+c and the bug happens >>


# umount: /mnt/mini/dev/pts: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)
umount: /mnt/mini/dev: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)




> As an easy check of the shell setup, could you run
> 
> zmodload zsh/parameter
> print $options[MONITOR]
> 
> in the problem case to see if job control has initialised properly?
> 
> pws
> 

Re: Bug: ZSH crashes upon receiving SIGINT

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 323 bytes --]

On Mar 11, 2015 4:30 PM, "John" <da_audiophile@yahoo.com> wrote:
>
> It seems OK.
>
> % zmodload zsh/parameter
>
> print $options[MONITOR]
> on
>
>
> % sudo arch-chroot /mnt/mini /bin/zsh# << this is in the chroot >>

Actually Peter wants you to run the zmodload and print from here inside the
chroot, not from outside it.

Re: Bug: ZSH crashes upon receiving SIGINT

[John]

>Actually Peter wants you to run the zmodload and print from here inside the chroot, not from outside it.


<<Headsmack>>

Running it from within the chroot gives the opposite output.  I googled for how I can toggle this parameter but didn't find a good answer.  Please advise.

# zmodload zsh/parameter
# print $options[MONITOR]
off

Re: Bug: ZSH crashes upon receiving SIGINT

[Andrew Janke]

Hi folks,

I was able to set up an Arch Linux instance that can consistently 
reproduce this issue. It's on a public-facing server, so if having 
access to work with this directly would help you debug it, send me a 
direct email with a desired username and public SSH key and I'll set you 
up with an account.


To set up a server to reproduce the issue:
Start with a fresh Arch Linux installation, and then:

pacman -Syy
pacman -S zsh arch-install-scripts
mkdir -p /chroots/mini
pacstrap -d -i /chroots/mini base zsh

Then to make the breakage happen:

arch-chroot /chroots/mini /bin/zsh

Wait for the prompt and hit Ctrl-C. It'll immediately produce those 
umount and input error messages.

After you get errors from this, your mount table will be in an unclean 
state and subsequent arch-chroot calls will fail. To clean up manually 
so you can arch-chroot again:

umount /chroots/mini/dev/pts
umount /chroots/mini/dev


Cheers,
Andrew

On 3/11/15 7:47 PM, John wrote:
>> Actually Peter wants you to run the zmodload and print from here inside the chroot, not from outside it.
>
> <<Headsmack>>
>
> Running it from within the chroot gives the opposite output.  I googled for how I can toggle this parameter but didn't find a good answer.  Please advise.
>
> # zmodload zsh/parameter
> # print $options[MONITOR]
> off

Re: Bug: ZSH crashes upon receiving SIGINT

[John]

> I was able to set up an Arch Linux instance that can consistently 
> reproduce this issue. It's on a public-facing server, so if having 
> access to work with this directly would help you debug it, send me a 
> direct email with a desired username and public SSH key and I'll set you 
> up with an account.


<SNIP>

Thanks for setting this up, Andrew.  I think it will be a great tool for some folks more knowledgeable then I to use to see this and diagnose the problem.

Re: Bug: ZSH crashes upon receiving SIGINT

[Peter Stephenson]

On Wed, 11 Mar 2015 23:47:07 +0000 (UTC)
John <da_audiophile@yahoo.com> wrote:
> # zmodload zsh/parameter
> # print $options[MONITOR]
> off

OK, that means job control isn't enabled.  Most likely it wasn't enabled
from the start because the shell found something about the environment
that didn't work.  That points towards Bart's explanation --- if
job control isn't enabled, zsh won't pick up the ^C by itself, it'll go
to whatever process group zsh was started in.

The final check for whether it can be enabled is in the init_io()
function and then acquire_pgrp().  It looks like the line editor is
working, otherwise it wouldn't be exiting at the point it is, so that
probably means SHTTY (a rather odd choice of variable for the file
descriptor used by the line editor and job control) is sane.

We might be able to confirm with some instrumentation as below...
Note this will be quite verbose if job control does get enabled
and lots of processes get run as it outputs a message when the
foreground process changes.

Bart may well have more idea what might be going wrong and have more
idea what to instrument, however.

pws

diff --git a/Src/init.c b/Src/init.c
index 3e41fb9..d49a176 100644
--- a/Src/init.c
+++ b/Src/init.c
@@ -591,15 +591,21 @@ init_io(void)
 	zsfree(ttystrname);
 	ttystrname = ztrdup("");
     } else {
+	fprintf(stderr, "SHTTY is open\n");
 #ifdef FD_CLOEXEC
 	long fdflags = fcntl(SHTTY, F_GETFD, 0);
 	if (fdflags != (long)-1) {
 	    fdflags |= FD_CLOEXEC;
 	    fcntl(SHTTY, F_SETFD, fdflags);
+	    fprintf(stderr, "Attempted to set FD_CLOEXEC on SHTTY\n");
 	}
 #endif
-	if (!ttystrname)
+	if (!ttystrname) {
 	    ttystrname = ztrdup("/dev/tty");
+	    fprintf(stderr, "Couldn't get TTY name\n");
+	} else {
+	    fprintf(stderr, "TTY name is %s\n", ttystrname);
+	}
     }
 
     /* We will only use zle if shell is interactive, *
@@ -619,11 +625,15 @@ init_io(void)
     if (opts[MONITOR] && (SHTTY != -1)) {
 	origpgrp = GETPGRP();
         acquire_pgrp(); /* might also clear opts[MONITOR] */
-    } else
+    } else {
+	if (opts[MONITOR])
+	    fprintf(stderr, "No terminal for job control\n");
 	opts[MONITOR] = 0;
+    }
 #else
     opts[MONITOR] = 0;
 #endif
+    fflush(stderr);
 }
 
 /**/
diff --git a/Src/jobs.c b/Src/jobs.c
index 295f4c9..6c0a4f7 100644
--- a/Src/jobs.c
+++ b/Src/jobs.c
@@ -2741,8 +2741,10 @@ acquire_pgrp(void)
 	sigaddset(&blockset, SIGTTOU);
 	sigaddset(&blockset, SIGTSTP);
 	oldset = signal_block(blockset);
+	fprintf(stderr, "Starting with mypgrp = %d\n", mypgrp);
 	while ((ttpgrp = gettygrp()) != -1 && ttpgrp != mypgrp) {
 	    mypgrp = GETPGRP();
+	    fprintf(stderr, "mypgrp -> %d\n", mypgrp);
 	    if (mypgrp == mypid) {
 		if (!interact)
 		    break; /* attachtty() will be a no-op, give up */
@@ -2764,12 +2766,17 @@ acquire_pgrp(void)
 	    if (setpgrp(0, 0) == 0) {
 		mypgrp = mypid;
 		attachtty(mypgrp);
-	    } else
+	    } else {
 		opts[MONITOR] = 0;
+		fprintf(stderr, "setpgrp failed, abandoning job control\n");
+	    }
 	}
 	signal_setmask(oldset);
-    } else
+    } else {
+	fprintf(stderr, "mypgrp = %d, abandoning job control\n",
+		mypgrp);
 	opts[MONITOR] = 0;
+    }
 }
 
 /* revert back to the process group we came from (before acquire_pgrp) */
diff --git a/Src/utils.c b/Src/utils.c
index 3d12807..83e5459 100644
--- a/Src/utils.c
+++ b/Src/utils.c
@@ -4148,6 +4148,7 @@ attachtty(pid_t pgrp)
     static int ep = 0;
 
     if (jobbing && interact) {
+	fprintf(stderr, "Attaching TTY %d to %d\n", SHTTY, pgrp); 
 #ifdef HAVE_TCSETPGRP
 	if (SHTTY != -1 && tcsetpgrp(SHTTY, pgrp) == -1 && !ep)
 #else
@@ -4160,13 +4161,16 @@ attachtty(pid_t pgrp)
 # endif
 #endif
 	{
-	    if (pgrp != mypgrp && kill(-pgrp, 0) == -1)
+	    if (pgrp != mypgrp && kill(-pgrp, 0) == -1) {
+		fprintf(stderr, "Bad process group, attaching to mypgrp\n");
 		attachtty(mypgrp);
-	    else {
+	    } else {
 		if (errno != ENOTTY)
 		{
 		    zwarn("can't set tty pgrp: %e", errno);
 		    fflush(stderr);
+		} else {
+		    fprintf(stderr, "Received ENOTTY, no job control\n");
 		}
 		opts[MONITOR] = 0;
 		ep = 1;

Re: Bug: ZSH crashes upon receiving SIGINT

[Bart Schaefer]

On Mar 12,  7:58pm, Peter Stephenson wrote:
} Subject: Re: Bug: ZSH crashes upon receiving SIGINT
}
} Bart may well have more idea what might be going wrong and have more
} idea what to instrument, however.

No, your instrumentation there looks pretty comprehensive to me.

Re: Bug: ZSH crashes upon receiving SIGINT

[Han Pingtian]

On Tue, Mar 10, 2015 at 05:43:27PM -0700, Bart Schaefer wrote:
> 
> It's *possible* that this is already "fixed" in development versions
> of zsh,

I tested according to what Dave suggested in the bug report, and looks
like it has ben fixed in development version:

% sudo unshare --fork --pid =zsh
[root@localhost]/tmp# print $ZSH_VERSION
5.0.7-dev-1
[root@localhost]/tmp# print $ZSH_PATCHLEVEL
zsh-5.0.7-336-ge85906e
[root@localhost]/tmp# <^C here>
[root@localhost]/tmp# <^C here>
[root@localhost]/tmp# <^C here>
[root@localhost]/tmp# <^C here>
[root@localhost]/tmp# <^C here>
[root@localhost]/tmp# <^D here>
%
% sudo unshare --fork --pid /usr/bin/zsh
[root@localhost]/tmp# print $ZSH_VERSION
5.0.7
[root@localhost]/tmp# <^C here>
[root@localhost]/tmp# %                                                                               %
[root@localhost]/tmp#
zsh: error on TTY read: Input/output error

%