Zsh scripts leave defunct processes when running under docker exec

Zsh scripts leave defunct processes when running under docker exec

[Soares Chen]

This only happens when the script is running inside a shell spawned by
`docker exec`. When running under normal system and under `docker
run`, there is no defunct process.

This bug is first discovered when I run nvm in docker
(https://github.com/creationix/nvm/issues/650). It was then discovered
that Docker's nsenter did not properly handle the SIGCHLD signal
raised from zsh. (https://github.com/docker/libcontainer/pull/369)
Even though it is now been fixed by Docker, I am not sure why zsh
would cause this bug in the first place.

I have a test script at the end of this post and also share it on
https://gist.github.com/soareschen/240e49116c7f2632d179

Steps to reproduce:

# terminal 1
docker run -it --rm --name test-zsh ubuntu:latest /bin/bash
apt-get install zsh curl
curl https://gist.githubusercontent.com/soareschen/240e49116c7f2632d179/raw/0be67acefd8d18fd62bb181998996f9a5772dc64/docker-zsh-test.sh
> docker-zsh-test.sh
chmod +x docker-zsh-test.sh
./docker-zsh-test.sh
ps auxf # no defunct process

# terminal 2
docker exec -it test-zsh /bin/zsh
./docker-zsh-test.sh
ps auxf # sed and zsh shown as defunct processes

#!/bin/zsh
# docker-zsh-test.sh
if [[ "$(echo "the black cat was chased by the brown dog" | sed -e
's/cat/fox/g' | grep fox)" = "" ]]; then
  echo "cat"
else
  echo "fox"
fi

Re: Zsh scripts leave defunct processes when running under docker exec

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

On Tuesday, February 10, 2015, Soares Chen <soares.chen@gmail.com> wrote:

> This only happens when the script is running inside a shell spawned by
> `docker exec`. When running under normal system and under `docker
> run`, there is no defunct process.
>
> This bug is first discovered when I run nvm in docker
> (https://github.com/creationix/nvm/issues/650). It was then discovered
> that Docker's nsenter did not properly handle the SIGCHLD signal
> raised from zsh. (https://github.com/docker/libcontainer/pull/369)
> Even though it is now been fixed by Docker, I am not sure why zsh
> would cause this bug in the first place.
>

(Someone more knowledgeable may be able to correct any misconceptions I
have about docker.)

I'm not exactly a docker expert, but my understanding is that docker
incorporates (is built on?) the relatively recent Linux feature of "PID
namespaces".  Under a PID namespace, the first process started is assigned
PID 1and is expected to behave like the "init" process in a traditional
Unix/Linux system, and be the ultimate manager of all the other processes
spawned within the namespace.

Zsh is not designed to assume that role.  In particular, when processing a
script, it will optimize process handling and perform an implicit "exec" of
the final command in the script, so that it need not hang around waiting
for that process to exit, assuming that it's own parent will reap it.  In a
"docker exec" there is no parent to do so.

Re: Zsh scripts leave defunct processes when running under docker exec

[Peter Stephenson]

On Tue, 10 Feb 2015 22:53:13 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> I'm not exactly a docker expert, but my understanding is that docker
> incorporates (is built on?) the relatively recent Linux feature of "PID
> namespaces".  Under a PID namespace, the first process started is assigned
> PID 1and is expected to behave like the "init" process in a traditional
> Unix/Linux system, and be the ultimate manager of all the other processes
> spawned within the namespace.
> 
> Zsh is not designed to assume that role.  In particular, when processing a
> script, it will optimize process handling and perform an implicit "exec" of
> the final command in the script, so that it need not hang around waiting
> for that process to exit, assuming that it's own parent will reap it.  In a
> "docker exec" there is no parent to do so.

Yeah, another special case we don't have either the bandwidth or the
expertise to support...

If that *is* it, is it just a case of special-casing PID 1 not to do
"fake" execs?  That's not so hard and shouldn't be problematic in other
cases.

pws

Re: Zsh scripts leave defunct processes when running under docker exec

[Bart Schaefer]

On Feb 11, 10:11am, Peter Stephenson wrote:
}
} If that *is* it, is it just a case of special-casing PID 1 not to do
} "fake" execs?  That's not so hard and shouldn't be problematic in other
} cases.

That would help, but I don't think that's everything, because zsh would
still not reap background grandchildren of children which exited first.

In fact I don't even know how it *could* do that; we've reached the limit
of my current knowledge of docker / PID namespaces.

Re: Zsh scripts leave defunct processes when running under docker exec

[Daniel Shahaf]

Bart Schaefer wrote on Wed, Feb 11, 2015 at 08:13:45 -0800:
> On Feb 11, 10:11am, Peter Stephenson wrote:
> }
> } If that *is* it, is it just a case of special-casing PID 1 not to do
> } "fake" execs?  That's not so hard and shouldn't be problematic in other
> } cases.
> 
> That would help, but I don't think that's everything, because zsh would
> still not reap background grandchildren of children which exited first.
> 
> In fact I don't even know how it *could* do that; we've reached the limit
> of my current knowledge of docker / PID namespaces.

Note that in the example, it is bash which runs as pid 1, not zsh.  The zsh
process running the script and the interactive one spawned by 'docker exec'
have pid != 1.

Re: Zsh scripts leave defunct processes when running under docker exec

[Bart Schaefer]

On Feb 13, 10:39am, Daniel Shahaf wrote:
} Subject: Re: Zsh scripts leave defunct processes when running under docker
}
} Bart Schaefer wrote on Wed, Feb 11, 2015 at 08:13:45 -0800:
} > That would help, but I don't think that's everything, because zsh would
} > still not reap background grandchildren of children which exited first.
} 
} Note that in the example, it is bash which runs as pid 1, not zsh.
} The zsh process running the script and the interactive one spawned by
} 'docker exec' have pid != 1.

I don't know if Chet is reading this and wants to chime in, but I suspect
bash is not expecting to take over the PID=1 role of init, either.

Re: Zsh scripts leave defunct processes when running under docker exec

[Soares Chen]

On Sun, Feb 15, 2015 at 2:58 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> I don't know if Chet is reading this and wants to chime in, but I suspect
> bash is not expecting to take over the PID=1 role of init, either.

I'm not sure how bash work differently from zsh, but my example script
works fine if changing the first line to run using bash.

I have talked to people at Docker. From my understanding, Linux has a
feature called subreaper that can take over the role of init and
become the parent of orphaned descendant processes. In this case  on
top of having different PID namespaces, `docker exec` is spawning a
process called nsenter to act as the subreaper. It is responsible to
reap defunct processes by handling SIGCHLD. With it being the
subreaper, all orphaned/defunct processes are captured by nsenter
instead of propogating to the actual init process inside the
container.

So current Docker didn't handle SIGCHLD properly and therefore causing
all orphaned processes to become defunct. It is probably not a problem
of zsh, just that its way of implicitly exec and orphaning child
processes in the final command causing this bug to become obvious in
Docker.

Thank you Bart for explaining and now I know a little better on how
zsh, Docker, and Linux works!

Soares

Re: Zsh scripts leave defunct processes when running under docker exec

[Peter Stephenson]

On Sun, 15 Feb 2015 22:03:05 +0800
Soares Chen <soares.chen@gmail.com> wrote:
> I have talked to people at Docker. From my understanding, Linux has a
> feature called subreaper that can take over the role of init and
> become the parent of orphaned descendant processes. In this case  on
> top of having different PID namespaces, `docker exec` is spawning a
> process called nsenter to act as the subreaper. It is responsible to
> reap defunct processes by handling SIGCHLD. With it being the
> subreaper, all orphaned/defunct processes are captured by nsenter
> instead of propogating to the actual init process inside the
> container.
> 
> So current Docker didn't handle SIGCHLD properly and therefore causing
> all orphaned processes to become defunct. It is probably not a problem
> of zsh, just that its way of implicitly exec and orphaning child
> processes in the final command causing this bug to become obvious in
> Docker.

Sounds entirely possible.  Thanks for tracking that down, it's very
useful to know.

pws