RE: runit not collecting zombies

["rehan khan" <rehan.khan@dsl.pipex.com>]

Actually a number of the updated packages *could* be the cause of this
issue. The package that I would look at closest is the bash upgrade as
runit scripts depend on the built-in exec command properly replacing the
shell with the same process id. Next see if the util-linux package
upgrade caused the problem. Last and most unlikely is the sysVinit
upgrade. I would guess downgrading one package at a time and checking
for zombies is the simplest methodology. I don't think the kernel is
implicated if you haven't restarted the machine. Again if I was to point
a finger it would be vaguely in the direction of the bash upgrade. As
bashtastic as it is it is not un-heard of for bash to have some quirks
version to version.

R

-----Original Message-----
From: Alex Efros [mailto:powerman@powerman.name] 
Sent: 25 February 2008 07:31
To: supervision@list.skarnet.org
Subject: Re: runit not collecting zombies

Hi!

I've just got new information. One of our servers doesn't have that
zombie
issue. But it wasn't updated for about 6 months because it previous
admin
retired. Now I begin updating it. At first I updated simple tools and
libraries, which shouldn't broke anything, and now I'm working on
updating
more critical things like toolchain, LAMP and few system tools like PAM
and udev. Current uptime is 128 days. And know what? I have just noticed
a
lot of zombies there! Usual trick 'chmod 0 /etc/runit/stopit; kill -CONT
1'
did the work, but... 

    I have neither updated runit (or other critical packages like
    toolchain) nor reboot system - this issue with zombies has arisen
    after update of some simple utils and libraries!!!!!
    WTF??? How this can be possible?!

If runit doesn't reload dynamic libraries on the fly or execute
external utils (which I believe is not the case), then how update of
some libraries and tools may result in runit failure to gather zombies?

Only possible "explanation" which come in my mind is some CPU/RAM usage
pattern which is happens while compiling all these packages somehow
affect
runit-init. And this isn't just "high load", because this server most of
time under high enough load, in all ways (CPU+RAM+HDD).

Below is list of updated packages, I can provide also versions "from" is
was updated, if somebody will need it.

Sat Feb 23 17:01:46 2008 >>> sys-apps/sandbox-1.2.18.1-r2
Sat Feb 23 17:02:21 2008 >>> sys-apps/portage-2.1.4.4
Sat Feb 23 21:48:03 2008 >>> app-misc/colordiff-1.0.6-r1
Sun Feb 24 02:17:13 2008 >>> sys-libs/ncurses-5.6-r2
Sun Feb 24 02:17:57 2008 >>> app-arch/bzip2-1.0.4-r1
Sun Feb 24 02:18:16 2008 >>> sys-devel/gnuconfig-20070724
Sun Feb 24 02:21:45 2008 >>> media-libs/freetype-2.3.5-r2
Sun Feb 24 02:24:49 2008 >>> dev-util/pkgconfig-0.22
Sun Feb 24 02:26:01 2008 >>> sys-libs/cracklib-2.8.12
Sun Feb 24 02:26:33 2008 >>> app-misc/pax-utils-0.1.16
Sun Feb 24 02:26:59 2008 >>> sys-devel/gcc-config-1.4.0-r4
Sun Feb 24 02:27:49 2008 >>> sys-libs/timezone-data-2007k
Sun Feb 24 02:29:33 2008 >>> media-libs/t1lib-5.0.2-r1
Sun Feb 24 02:32:08 2008 >>> dev-libs/libmcrypt-2.5.8
Sun Feb 24 02:35:28 2008 >>> dev-db/sqlite-3.5.3
Sun Feb 24 02:36:52 2008 >>> sys-fs/sysfsutils-2.1.0
Sun Feb 24 02:37:13 2008 >>> net-analyzer/netselect-0.3-r2
Sun Feb 24 02:37:30 2008 >>> sys-process/cronbase-0.3.2-r1
Sun Feb 24 02:40:02 2008 >>> dev-lang/spidermonkey-1.7.0
Sun Feb 24 02:42:02 2008 >>> app-arch/cpio-2.9-r1
Sun Feb 24 02:42:43 2008 >>> sys-process/acct-6.3.5-r2
Sun Feb 24 02:43:16 2008 >>> app-portage/portage-utils-0.1.29
Sun Feb 24 02:56:51 2008 >>> app-arch/p7zip-4.57
Sun Feb 24 02:58:10 2008 >>> sys-fs/reiserfsprogs-3.6.19-r2
Sun Feb 24 02:58:53 2008 >>> net-proxy/3proxy-0.5.3j
Sun Feb 24 02:59:19 2008 >>> net-analyzer/traceroute-2.0.9-r1
Sun Feb 24 03:01:08 2008 >>> sys-kernel/linux-headers-2.6.23-r3
Sun Feb 24 03:04:43 2008 >>> app-shells/bash-3.2_p17-r1
Sun Feb 24 03:07:02 2008 >>> media-libs/libpng-1.2.24
Sun Feb 24 03:09:36 2008 >>> dev-libs/libpcre-7.6-r1
Sun Feb 24 03:13:22 2008 >>> media-libs/tiff-3.8.2-r3
Sun Feb 24 03:14:18 2008 >>> sys-apps/less-418
Sun Feb 24 03:24:46 2008 >>> app-portage/eix-0.10.2
Sun Feb 24 03:26:09 2008 >>> sys-apps/pciutils-2.2.9
Sun Feb 24 03:27:51 2008 >>> net-misc/netkit-telnetd-0.17-r8
Sun Feb 24 03:34:33 2008 >>> sys-libs/readline-5.2_p12-r1
Sun Feb 24 03:36:50 2008 >>> sys-apps/sysvinit-2.86-r10
Sun Feb 24 03:38:53 2008 >>> sys-apps/ed-0.8
Sun Feb 24 03:41:09 2008 >>> net-misc/iputils-20070202
Sun Feb 24 04:06:57 2008 >>> dev-libs/libxml2-2.6.30-r1
Sun Feb 24 04:53:51 2008 >>> sys-devel/gettext-0.17
Sun Feb 24 04:55:04 2008 >>> sys-apps/sed-4.1.5
Sun Feb 24 04:56:47 2008 >>> sys-devel/m4-1.4.10
Sun Feb 24 04:57:34 2008 >>> sys-apps/man-1.6f
Sun Feb 24 04:58:44 2008 >>> sys-devel/flex-2.5.33-r3
Sun Feb 24 04:59:48 2008 >>> dev-libs/libgpg-error-1.6
Sun Feb 24 05:02:31 2008 >>> sys-apps/findutils-4.3.11
Sun Feb 24 05:04:23 2008 >>> sys-apps/gawk-3.1.5-r5
Sun Feb 24 05:05:38 2008 >>> app-editors/nano-2.0.7
Sun Feb 24 05:06:49 2008 >>> dev-util/dialog-1.1.20071028
Sun Feb 24 05:07:51 2008 >>> sys-apps/kbd-1.13-r1
Sun Feb 24 05:10:45 2008 >>> app-arch/tar-1.19-r1
Sun Feb 24 05:11:23 2008 >>> sys-apps/ucspi-tcp-0.88-r16
Sun Feb 24 05:12:49 2008 >>> sys-apps/man-pages-2.76
Sun Feb 24 05:13:23 2008 >>> sys-devel/bc-1.06-r6
Sun Feb 24 05:13:34 2008 >>> virtual/editor-0
Sun Feb 24 05:26:40 2008 >>> sys-devel/binutils-2.18-r1
Sun Feb 24 05:28:49 2008 >>> sys-libs/com_err-1.40.4
Sun Feb 24 05:41:56 2008 >>> sys-libs/db-4.5.20_p2
Sun Feb 24 05:43:16 2008 >>> sys-libs/ss-1.40.4
Sun Feb 24 05:43:38 2008 >>> sys-apps/paxctl-0.5
Sun Feb 24 05:46:54 2008 >>> sys-fs/e2fsprogs-1.40.4
Sun Feb 24 05:49:59 2008 >>> sys-apps/util-linux-2.13-r2
Sun Feb 24 05:54:34 2008 >>> sys-apps/parted-1.8.8
Sun Feb 24 05:56:03 2008 >>> net-mail/fetchmail-6.3.8-r1
Sun Feb 24 06:01:18 2008 >>> net-misc/ntp-4.2.4_p4
Sun Feb 24 06:15:26 2008 >>> dev-lang/perl-5.8.8-r4
Sun Feb 24 06:16:04 2008 >>> sys-devel/autoconf-2.61-r1
Sun Feb 24 06:16:20 2008 >>> app-admin/perl-cleaner-1.05
Sun Feb 24 06:18:05 2008 >>> dev-libs/libtasn1-1.2
Sun Feb 24 06:18:25 2008 >>> sys-apps/help2man-1.36.4
Sun Feb 24 06:20:09 2008 >>> net-misc/rsync-2.6.9-r5
Sun Feb 24 06:22:37 2008 >>> sys-devel/libtool-1.5.26
Sun Feb 24 07:56:47 2008 >>> sys-apps/busybox-1.8.2
Sun Feb 24 07:57:13 2008 >>> sys-apps/slocate-3.1-r1
Sun Feb 24 07:58:20 2008 >>> net-libs/libpcap-0.9.8
Sun Feb 24 08:02:15 2008 >>> dev-libs/libgcrypt-1.4.0-r1
Sun Feb 24 08:03:07 2008 >>> dev-perl/DBI-1.601
Sun Feb 24 08:03:47 2008 >>> x11-misc/read-edid-1.4.1-r1
Sun Feb 24 08:05:52 2008 >>> net-dns/libidn-1.0-r1
Sun Feb 24 08:07:13 2008 >>> sys-process/psmisc-22.6
Sun Feb 24 08:28:13 2008 >>> net-fs/samba-3.0.28
Sun Feb 24 08:30:00 2008 >>> dev-util/strace-4.5.16
Sun Feb 24 08:35:23 2008 >>> net-analyzer/rrdtool-1.2.23-r1
Sun Feb 24 08:38:41 2008 >>> dev-libs/libxslt-1.1.22
Sun Feb 24 08:40:32 2008 >>> app-crypt/opencdk-0.6.6
Sun Feb 24 08:41:20 2008 >>> net-analyzer/lft-3.0
Sun Feb 24 08:45:39 2008 >>> net-analyzer/nmap-4.53
Sun Feb 24 08:46:01 2008 >>> net-misc/whois-4.7.24
Sun Feb 24 08:47:51 2008 >>> net-analyzer/tcpdump-3.9.8
Sun Feb 24 08:57:19 2008 >>> net-libs/gnutls-2.0.4
Sun Feb 24 09:01:41 2008 >>> net-misc/curl-7.17.1
Sun Feb 24 09:08:55 2008 >>> dev-lang/python-2.4.4-r6
Sun Feb 24 09:10:28 2008 >>> sys-apps/file-4.23
Sun Feb 24 09:10:44 2008 >>> app-admin/eselect-vi-1.1.5
Sun Feb 24 09:10:55 2008 >>> app-admin/eselect-ctags-1.3
Sun Feb 24 09:11:55 2008 >>> dev-util/ctags-5.7
Sun Feb 24 09:12:25 2008 >>> sys-apps/debianutils-2.28.2
Sun Feb 24 09:12:45 2008 >>> app-portage/gentoolkit-0.2.3-r1
Sun Feb 24 09:13:18 2008 >>> sys-apps/baselayout-1.12.11.1
Sun Feb 24 09:14:03 2008 >>> sys-apps/module-init-tools-3.4
Sun Feb 24 09:30:04 2008 >>> sys-kernel/hardened-sources-2.6.23-r7
Sun Feb 24 09:31:40 2008 >>> net-dialup/ppp-2.4.4-r14
Sun Feb 24 09:33:01 2008 >>> sys-apps/lm_sensors-2.10.4
Sun Feb 24 09:34:21 2008 >>> net-firewall/iptables-1.3.8-r3
Sun Feb 24 09:36:33 2008 >>> app-editors/vim-core-7.1.123
Sun Feb 24 09:41:04 2008 >>> app-editors/vim-7.1.123
Sun Feb 24 10:08:14 2008 >>> dev-lang/php-5.2.4_pre200708051230-r2
Sun Feb 24 10:09:42 2008 >>> dev-php/PEAR-PEAR-1.6.2-r1
Sun Feb 24 10:09:53 2008 >>> app-admin/eselect-fontconfig-1.0
Sun Feb 24 10:13:11 2008 >>> media-libs/fontconfig-2.5.0-r1
Sun Feb 24 10:13:49 2008 >>> media-fonts/corefonts-1-r4
Sun Feb 24 10:35:59 2008 >>> media-gfx/imagemagick-6.3.7.9
Sun Feb 24 11:02:04 2008 >>> dev-libs/glib-2.14.6
Sun Feb 24 11:02:30 2008 >>> app-arch/rar-3.7.1

These packages was installed, not updated:
Sun Feb 24 18:10:56 2008 >>> app-misc/mc-mp-4.1.40_pre9-r1
Sun Feb 24 23:47:46 2008 >>> app-forensics/chkrootkit-0.47
Sun Feb 24 23:48:45 2008 >>> sys-process/lsof-4.78-r1
Sun Feb 24 23:49:00 2008 >>> app-forensics/rkhunter-1.2.9

P.S. Gerrit: runit is really cool, but this bug (unfixed for about a
year!)
drives me crazy... :(

-- 
			WBR, Alex.