Re: runit not collecting zombies

[Charlie Brady <charlieb-supervision@budge.apana.org.au>]

On Wed, 12 Sep 2007, Alex Efros wrote:

> On Wed, Sep 12, 2007 at 01:40:07PM -0400, Charlie Brady wrote:
>> Indeed. Please remember that we haven't seen your ps output.
>
> Oh, really? How about this one:
> http://article.gmane.org/gmane.comp.sysutils.supervision.general/1422
> and this:
> http://article.gmane.org/gmane.comp.sysutils.supervision.general/1482

Yep, you've got me there.

>> If the parent sshd continues to run, then it can fork lots of children, all
>> or many of which exit very quickly, and there will still be no zombies
>> reparented to init. There's something more going on here. You would be well
>> advised to report the problem to whoever maintains the ssh which you use
>> and/or the ssh maintainers.
>
> Hmm. This sounds reasonable enough, I haven't think about this.
> Actually, parent ssh never exit - I never see /var/service/ssh/run restarted!

OK, so that means that any zombie process must be at least a child of a 
child.

If we look at this:

...
sshd      2421     1  0 Jul14 ?        Z      0:00 [sshd] <defunct>
sshd      2423     1  0 Jul14 ?        Z      0:00 [sshd] <defunct>
sshd      2425     1  0 Jul14 ?        Z      0:00 [sshd] <defunct>
sshd      2427     1  0 Jul14 ?        Z      0:00 [sshd] <defunct>
sshd      2429     1  0 Jul14 ?        Z      0:00 [sshd] <defunct>
sshd      2431     1  0 Jul14 ?        Z      0:00 [sshd] <defunct>
...

you'll see every second pid is a zombie. This could occur if the ancestor 
sshd forks, then the child forks again, and the parent of the grandchild 
exits without waiting for its child. Once the child exits, it will be a 
zombie until process 1 reaps its status.

In the example shown, let's make the ancestor sshd process 100. Then it 
forks and produces process 2420. 2420 forks to produce 2421, then exits. 
100 reaps the exit status of 2420, so 2420 disappears from the process 
table. Then 2421 exits, and appears as a zombie until its status is reaped 
by proc 1.

100 forks again and produces process 4222. 4222 forks to produce 2423, 
then exits. 100 reaps the exit status of 2422, so 2422 disappears from the 
process table. Then 2423 exits, and appears as a zombie until its status 
is reaped by proc 1.

etc.

The ssh maintainers should be interested in your process table.

If you mention what version you are running, someone might be interested 
to go looking through the ssh code to find out how the scenario you show 
could have occurred. An strace which captures the fork/fork/exit sequence 
as it happens would be very useful.

---
Charlie