Re: runit not collecting zombies

[Mike Buland <mike@geekgene.com>]

Quite honestly, I don't know which ones would or wouldn't.  It doesn't seem 
likely, but it's the only thing that you've mentioned yet that seems like it 
could be contributing.  If I were going to try to find this issue, I would 
start with an older set of packages, something from hardened gentoo from back 
when you say it worked, from April or so (I guess), and see if I can 
reproduce this problem.  Upgrade packages and see what affects runit.

Yes, they're all live servers and you can't stand to have them down or 
non-hardened, and that's fine.  Pull an old desktop and use it, just for some 
testing, you shouldn't need to wait too long if you can get anything to 
produce zombies, and there you go.  If you can get closer to pinpointing what 
on your list may have an effect, then it will be easier to solve.

Good Luck,
--Mike

On Wednesday 12 September 2007 02:28:42 pm Alex Efros wrote:
> Hi!
>
> On Wed, Sep 12, 2007 at 01:38:54PM -0600, Mike Buland wrote:
> > I'm just curious, but doesn't it sound like this is the first place to
> > look for the trouble?  Unfortunately, as you point out, there are two
> > differences between the two systems, the one that works isn't using two
> > of the hardened patches, and is using a newer gcc.  Have you reported
> > these facts to the
>
> Yep. I dislike both idea to use non-hardened gcc on production servers and
> even more dislike idea to upgrade to gcc-4.1.1 without ability to safely
> disgrade after testing. Remember, this issue happens every ~week, so I
> should wait at least 3 weeks because saying 'huh, changing gcc solved
> issue'.
>
> I've tried to analyze this from other side. As I noted here:
>     http://bugs.gentoo.org/show_bug.cgi?id=190261#c1
> this issue happens at some time, and then repeated every 2-10 days.
> So, looks like something was changed on all my servers 2-10 days BEFORE
> this issue happens for the first time. Only changed thing was usual
> upgrade for some Gentoo packages. And I know when this issue happens for
> me first time: 2007-05-26. And I've logs for all package upgrades and
> server reboots for that period:
>
> Fri Apr 21 19:18:39 2006 >>> sys-process/runit-1.5.0
> ...
> Kernel 2.6.16-hardened-r11 was used from Sep 10 12:46:47 GMT 2006
> ...
> Sun Sep 10 17:42:41 2006 >>> sys-devel/gcc-3.4.6-r1
> ...
> Mon Dec 18 02:25:38 2006 >>> sys-libs/glibc-2.3.6-r5
> ...
> reboot (2.6.16-hardened-r11) at Sat Dec 23 23:58:49 GMT 2006
> ...
> Mon Jan  1 21:35:05 2007 >>> sys-devel/gcc-3.4.6-r2
> ...
> Sat Mar 31 01:45:24 2007 >>> sys-devel/gcc-3.4.6-r2
> ...
> Sun Apr  1 13:37:43 2007 >>> dev-lang/perl-5.8.8-r2
> Sun Apr  1 13:41:18 2007 >>> dev-lang/perl-5.8.8-r2
> Sun Apr  1 13:41:49 2007 >>> dev-perl/Net-Daemon-0.39
> Sun Apr  1 13:41:54 2007 >>> dev-perl/PlRPC-0.2018
> Sun Apr  1 13:42:09 2007 >>> dev-perl/DBI-1.53
> Sun Apr  1 13:42:26 2007 >>> dev-perl/DBD-mysql-3.0008
> Sun Apr  1 17:59:45 2007 >>> app-misc/mime-types-7
> Sun Apr  1 18:00:57 2007 >>> sys-apps/man-1.6e-r1
> Sun Apr  1 18:07:55 2007 >>> sys-libs/db-4.3.29-r2
> Sun Apr  1 18:08:07 2007 >>> app-portage/gentoolkit-0.2.3-r1
> Sun Apr  8 18:12:28 2007 >>> sys-libs/ncurses-5.6
> Sun Apr  8 18:13:15 2007 >>> sys-apps/file-4.20-r1
> Wed Apr 11 03:08:33 2007 >>> sys-apps/man-pages-2.44
> reboot (2.6.16-hardened-r11) at Fri Apr 27 21:55:13 GMT 2007
> Sun May  6 19:05:48 2007 >>> sys-apps/debianutils-2.17.5
> Sun May  6 19:08:07 2007 >>> dev-libs/apr-0.9.12
> Sun May  6 19:11:34 2007 >>> dev-util/pkgconfig-0.21-r1
> Sun May  6 19:11:54 2007 >>> sys-libs/timezone-data-2007d
> Sun May  6 19:12:48 2007 >>> dev-lang/spidermonkey-1.5-r2
> Sun May  6 19:13:17 2007 >>> sys-devel/patch-2.5.9-r1
> Sun May  6 19:13:24 2007 >>> sys-apps/hdparm-6.9-r1
> Sun May  6 19:14:28 2007 >>> net-misc/rsync-2.6.9-r2
> Sun May  6 19:15:34 2007 >>> dev-libs/pth-2.0.6
> Sun May  6 19:15:37 2007 >>> sys-devel/binutils-config-1.9-r4
> Sun May  6 19:19:57 2007 >>> app-shells/bash-3.2_p15-r1
> Sun May  6 19:20:31 2007 >>> dev-util/dialog-1.1.20070227
> Sun May  6 19:20:59 2007 >>> sys-apps/man-1.6e-r3
> Sun May  6 19:22:14 2007 >>> media-libs/libpng-1.2.16
> Sun May  6 19:23:48 2007 >>> media-libs/freetype-2.1.10-r3
> Sun May  6 19:23:58 2007 >>> app-misc/ca-certificates-20070303-r1
> Sun May  6 19:26:04 2007 >>> sys-libs/readline-5.2_p2
> Sun May  6 19:27:49 2007 >>> dev-libs/libgpg-error-1.5
> Sun May  6 19:28:43 2007 >>> sys-devel/m4-1.4.9
> Sun May  6 19:30:40 2007 >>> sys-fs/e2fsprogs-1.39-r2
> Sun May  6 19:31:19 2007 >>> app-editors/nano-2.0.4
> Sun May  6 19:32:19 2007 >>> net-mail/fetchmail-6.3.8
> Sun May  6 19:32:55 2007 >>> sys-devel/flex-2.5.33-r2
> Sun May  6 19:33:25 2007 >>> sys-apps/baselayout-1.12.9-r2
> Sun May  6 19:36:02 2007 >>> sys-apps/util-linux-2.12r-r6
> Sun May  6 19:37:14 2007 >>> app-editors/vim-core-7.0.235
> Sun May  6 19:38:27 2007 >>> dev-libs/libksba-1.0.0
> Sun May  6 19:40:22 2007 >>> dev-libs/libxslt-1.1.20
> Sun May  6 19:41:04 2007 >>> sys-apps/module-init-tools-3.2.2-r3
> Sun May  6 19:47:27 2007 >>> app-editors/vim-7.0.235
> Sun May  6 19:53:14 2007 >>> sys-kernel/hardened-sources-2.6.20-r2
> Sun May  6 19:56:14 2007 >>> net-misc/curl-7.15.1-r1
> Sun May  6 20:36:14 2007 >>> dev-db/mysql-5.0.38
> Sun May  6 20:53:01 2007 >>> media-gfx/imagemagick-6.3.3
> Sun May  6 20:53:19 2007 >>> sys-devel/gcc-config-1.3.16
> Sun May  6 21:11:35 2007 >>> sys-libs/libstdc++-v3-3.3.6
> Wed May  9 15:53:39 2007 >>> media-libs/freetype-2.3.3
> Wed May  9 15:59:54 2007 >>> dev-lang/python-2.4.4
> Wed May  9 16:04:33 2007 >>> mail-mta/netqmail-1.05-r8
> Wed May 23 13:51:19 2007 >>> sys-apps/portage-2.1.2.7
> Wed May 23 13:51:45 2007 >>> sys-libs/timezone-data-2007e
> Wed May 23 13:51:55 2007 >>> app-forensics/chkrootkit-0.47
> Wed May 23 13:52:28 2007 >>> sys-libs/zlib-1.2.3-r1
> Wed May 23 13:53:50 2007 >>> media-libs/libpng-1.2.18
> Wed May 23 13:56:07 2007 >>> media-libs/freetype-2.3.4-r2
> Wed May 23 14:44:45 2007 >>> dev-db/mysql-5.0.40
> Wed May 23 14:50:22 2007 >>> dev-lang/python-2.4.4-r4
> Wed May 23 14:50:24 2007 >>> app-admin/python-updater-0.2
> Wed May 23 14:51:36 2007 >>> sys-apps/util-linux-2.12r-r7
> Wed May 23 14:52:04 2007 >>> sys-apps/gradm-2.1.10.200702231759
> Wed May 23 14:59:26 2007 >>> app-crypt/gnupg-1.4.7-r1
> reboot (2.6.16-hardened-r11) at Sat May 26 10:37:41 GMT 2007
> reboot (2.6.16-hardened-r11) at Sat Jun  2 14:54:20 GMT 2007
> Sun Jun  3 13:04:10 2007 >>> app-portage/eix-0.9.1
> reboot (2.6.16-hardened-r11) at Sat Jun  9 14:56:38 GMT 2007
> Mon Jun 11 13:00:46 2007 >>> sys-process/runit-1.7.2
> reboot (2.6.16-hardened-r11) at Mon Jun 11 13:06:48 GMT 2007
> reboot (2.6.16-hardened-r11) at Sat Jun 16 03:59:03 GMT 2007
> reboot (2.6.20-hardened-r2)  at Sat Jun 16 04:26:30 GMT 2007
> ...
> Thu Jun 14 23:12:38 2007 >>> sys-libs/glibc-2.5-r3
>
> Neither runit, nor gcc, glibc or kernel was upgraded in May (kernel
> sources was unpacked in /usr/src at May 6, but it was compiled and boot
> only Jun 16).
>
> Any ideas how upgrading THESE packages may affect zombie reaping in
> already running runit? :-/