Re: runit not collecting zombies

[Alex Efros <powerman@powerman.asdfGroup.com>]

Hi!

On Tue, Jun 19, 2007 at 06:13:25PM +0000, Gerrit Pape wrote:
> Hi Alex, after checking the code, I currently cannot say that or how
> runit could fail reaping zombies that detached and re-parented to pid 1.
> On Linux running strace on pid 1 isn't supported AFAIK.  To be sure that
> runit is at fault, can you please check the kernel versions on your two
> machines, can it be that they have changed at the time the problem
> popped up?  Does upgrading the Linux kernel to a more recent version
> change anything?

Yeah, I also think this may be kernel<->runit bug. But this bug happens
previously on 2.6.16, and now I've it on 2.6.20 (upgraded few days ago).

Moreover, looks like runit not just 'stop reaping zombies' at some point.
Looks like it continue reaping them, but not all of them. Look:

# date; ps ax | grep Z | wc
Mon Jun 18 13:46:02 GMT 2007
    162     973    7155
# date; ps ax | grep Z | wc
Mon Jun 18 22:41:58 GMT 2007
    406    2437   17894
# date; ps ax | grep Z | wc
Tue Jun 19 18:59:46 GMT 2007
    770    4621   33939

This server generate a lot of short-living processes every minute, i.e. it
generate new processes much faster than new non-reaped zombies arise.

I may be wrong, but I've a feeling on 2.6.16 new non-reaped zombies arise
much faster - in few hours I got 8192 use processes and was forced to reboot.

Maybe kernel just don't send SIGCHLD in some situation? Maybe some race
condition in runit or kernel? Maybe if runit will try to do waitpid()
every few seconds this will solve issue?

-- 
			WBR, Alex.