Re: runit not collecting zombies

[Alex Efros <powerman@powerman.asdfGroup.com>]

Hi!

On Wed, Sep 12, 2007 at 01:38:54PM -0600, Mike Buland wrote:
> I'm just curious, but doesn't it sound like this is the first place to look 
> for the trouble?  Unfortunately, as you point out, there are two differences 
> between the two systems, the one that works isn't using two of the hardened 
> patches, and is using a newer gcc.  Have you reported these facts to the 

Yep. I dislike both idea to use non-hardened gcc on production servers and
even more dislike idea to upgrade to gcc-4.1.1 without ability to safely
disgrade after testing. Remember, this issue happens every ~week, so I
should wait at least 3 weeks because saying 'huh, changing gcc solved issue'.

I've tried to analyze this from other side. As I noted here:
    http://bugs.gentoo.org/show_bug.cgi?id=190261#c1
this issue happens at some time, and then repeated every 2-10 days.
So, looks like something was changed on all my servers 2-10 days BEFORE
this issue happens for the first time. Only changed thing was usual
upgrade for some Gentoo packages. And I know when this issue happens for
me first time: 2007-05-26. And I've logs for all package upgrades and server
reboots for that period:

Fri Apr 21 19:18:39 2006 >>> sys-process/runit-1.5.0
...
Kernel 2.6.16-hardened-r11 was used from Sep 10 12:46:47 GMT 2006
...
Sun Sep 10 17:42:41 2006 >>> sys-devel/gcc-3.4.6-r1
...
Mon Dec 18 02:25:38 2006 >>> sys-libs/glibc-2.3.6-r5
...
reboot (2.6.16-hardened-r11) at Sat Dec 23 23:58:49 GMT 2006
...
Mon Jan  1 21:35:05 2007 >>> sys-devel/gcc-3.4.6-r2
...
Sat Mar 31 01:45:24 2007 >>> sys-devel/gcc-3.4.6-r2
...
Sun Apr  1 13:37:43 2007 >>> dev-lang/perl-5.8.8-r2
Sun Apr  1 13:41:18 2007 >>> dev-lang/perl-5.8.8-r2
Sun Apr  1 13:41:49 2007 >>> dev-perl/Net-Daemon-0.39
Sun Apr  1 13:41:54 2007 >>> dev-perl/PlRPC-0.2018
Sun Apr  1 13:42:09 2007 >>> dev-perl/DBI-1.53
Sun Apr  1 13:42:26 2007 >>> dev-perl/DBD-mysql-3.0008
Sun Apr  1 17:59:45 2007 >>> app-misc/mime-types-7
Sun Apr  1 18:00:57 2007 >>> sys-apps/man-1.6e-r1
Sun Apr  1 18:07:55 2007 >>> sys-libs/db-4.3.29-r2
Sun Apr  1 18:08:07 2007 >>> app-portage/gentoolkit-0.2.3-r1
Sun Apr  8 18:12:28 2007 >>> sys-libs/ncurses-5.6
Sun Apr  8 18:13:15 2007 >>> sys-apps/file-4.20-r1
Wed Apr 11 03:08:33 2007 >>> sys-apps/man-pages-2.44
reboot (2.6.16-hardened-r11) at Fri Apr 27 21:55:13 GMT 2007
Sun May  6 19:05:48 2007 >>> sys-apps/debianutils-2.17.5
Sun May  6 19:08:07 2007 >>> dev-libs/apr-0.9.12
Sun May  6 19:11:34 2007 >>> dev-util/pkgconfig-0.21-r1
Sun May  6 19:11:54 2007 >>> sys-libs/timezone-data-2007d
Sun May  6 19:12:48 2007 >>> dev-lang/spidermonkey-1.5-r2
Sun May  6 19:13:17 2007 >>> sys-devel/patch-2.5.9-r1
Sun May  6 19:13:24 2007 >>> sys-apps/hdparm-6.9-r1
Sun May  6 19:14:28 2007 >>> net-misc/rsync-2.6.9-r2
Sun May  6 19:15:34 2007 >>> dev-libs/pth-2.0.6
Sun May  6 19:15:37 2007 >>> sys-devel/binutils-config-1.9-r4
Sun May  6 19:19:57 2007 >>> app-shells/bash-3.2_p15-r1
Sun May  6 19:20:31 2007 >>> dev-util/dialog-1.1.20070227
Sun May  6 19:20:59 2007 >>> sys-apps/man-1.6e-r3
Sun May  6 19:22:14 2007 >>> media-libs/libpng-1.2.16
Sun May  6 19:23:48 2007 >>> media-libs/freetype-2.1.10-r3
Sun May  6 19:23:58 2007 >>> app-misc/ca-certificates-20070303-r1
Sun May  6 19:26:04 2007 >>> sys-libs/readline-5.2_p2
Sun May  6 19:27:49 2007 >>> dev-libs/libgpg-error-1.5
Sun May  6 19:28:43 2007 >>> sys-devel/m4-1.4.9
Sun May  6 19:30:40 2007 >>> sys-fs/e2fsprogs-1.39-r2
Sun May  6 19:31:19 2007 >>> app-editors/nano-2.0.4
Sun May  6 19:32:19 2007 >>> net-mail/fetchmail-6.3.8
Sun May  6 19:32:55 2007 >>> sys-devel/flex-2.5.33-r2
Sun May  6 19:33:25 2007 >>> sys-apps/baselayout-1.12.9-r2
Sun May  6 19:36:02 2007 >>> sys-apps/util-linux-2.12r-r6
Sun May  6 19:37:14 2007 >>> app-editors/vim-core-7.0.235
Sun May  6 19:38:27 2007 >>> dev-libs/libksba-1.0.0
Sun May  6 19:40:22 2007 >>> dev-libs/libxslt-1.1.20
Sun May  6 19:41:04 2007 >>> sys-apps/module-init-tools-3.2.2-r3
Sun May  6 19:47:27 2007 >>> app-editors/vim-7.0.235
Sun May  6 19:53:14 2007 >>> sys-kernel/hardened-sources-2.6.20-r2
Sun May  6 19:56:14 2007 >>> net-misc/curl-7.15.1-r1
Sun May  6 20:36:14 2007 >>> dev-db/mysql-5.0.38
Sun May  6 20:53:01 2007 >>> media-gfx/imagemagick-6.3.3
Sun May  6 20:53:19 2007 >>> sys-devel/gcc-config-1.3.16
Sun May  6 21:11:35 2007 >>> sys-libs/libstdc++-v3-3.3.6
Wed May  9 15:53:39 2007 >>> media-libs/freetype-2.3.3
Wed May  9 15:59:54 2007 >>> dev-lang/python-2.4.4
Wed May  9 16:04:33 2007 >>> mail-mta/netqmail-1.05-r8
Wed May 23 13:51:19 2007 >>> sys-apps/portage-2.1.2.7
Wed May 23 13:51:45 2007 >>> sys-libs/timezone-data-2007e
Wed May 23 13:51:55 2007 >>> app-forensics/chkrootkit-0.47
Wed May 23 13:52:28 2007 >>> sys-libs/zlib-1.2.3-r1
Wed May 23 13:53:50 2007 >>> media-libs/libpng-1.2.18
Wed May 23 13:56:07 2007 >>> media-libs/freetype-2.3.4-r2
Wed May 23 14:44:45 2007 >>> dev-db/mysql-5.0.40
Wed May 23 14:50:22 2007 >>> dev-lang/python-2.4.4-r4
Wed May 23 14:50:24 2007 >>> app-admin/python-updater-0.2
Wed May 23 14:51:36 2007 >>> sys-apps/util-linux-2.12r-r7
Wed May 23 14:52:04 2007 >>> sys-apps/gradm-2.1.10.200702231759
Wed May 23 14:59:26 2007 >>> app-crypt/gnupg-1.4.7-r1
reboot (2.6.16-hardened-r11) at Sat May 26 10:37:41 GMT 2007
reboot (2.6.16-hardened-r11) at Sat Jun  2 14:54:20 GMT 2007
Sun Jun  3 13:04:10 2007 >>> app-portage/eix-0.9.1
reboot (2.6.16-hardened-r11) at Sat Jun  9 14:56:38 GMT 2007
Mon Jun 11 13:00:46 2007 >>> sys-process/runit-1.7.2
reboot (2.6.16-hardened-r11) at Mon Jun 11 13:06:48 GMT 2007
reboot (2.6.16-hardened-r11) at Sat Jun 16 03:59:03 GMT 2007
reboot (2.6.20-hardened-r2)  at Sat Jun 16 04:26:30 GMT 2007
...
Thu Jun 14 23:12:38 2007 >>> sys-libs/glibc-2.5-r3

Neither runit, nor gcc, glibc or kernel was upgraded in May (kernel
sources was unpacked in /usr/src at May 6, but it was compiled and boot
only Jun 16).

Any ideas how upgrading THESE packages may affect zombie reaping in
already running runit? :-/

-- 
			WBR, Alex.