From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.sysutils.supervision.general/1651 Path: news.gmane.org!not-for-mail From: "rehan khan" Newsgroups: gmane.comp.sysutils.supervision.general Subject: RE: runit not collecting zombies Date: Mon, 25 Feb 2008 16:24:22 -0000 Message-ID: <50F2BE60A0EF6D478B1BCC633DEC28CC082BEE@server.home.internal> References: <46561ABE.7030008@podgorny.cz> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1203956678 20875 80.91.229.12 (25 Feb 2008 16:24:38 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 25 Feb 2008 16:24:38 +0000 (UTC) To: "Alex Efros" , Original-X-From: supervision-return-1886-gcsg-supervision=m.gmane.org@list.skarnet.org Mon Feb 25 17:25:03 2008 Return-path: Envelope-to: gcsg-supervision@gmane.org Original-Received: from antah.skarnet.org ([212.85.147.14]) by lo.gmane.org with smtp (Exim 4.50) id 1JTg8P-000312-16 for gcsg-supervision@gmane.org; Mon, 25 Feb 2008 17:24:57 +0100 Original-Received: (qmail 28392 invoked by uid 76); 25 Feb 2008 16:24:47 -0000 Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Archive: Original-Received: (qmail 28386 invoked from network); 25 Feb 2008 16:24:47 -0000 Original-X-Trace: 48226182/mk-outboundfilter-1.mail.uk.tiscali.com/PIPEX/$ACCEPTED/pipex-customers/81.86.133.231 X-SBRS: None X-RemoteIP: 81.86.133.231 X-IP-MAIL-FROM: rehan.khan@dsl.pipex.com X-IP-BHB: Once X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ao8CAKx6wkdRVoXn/2dsb2JhbACrRw X-IP-Direction: IN In-Reply-To: Content-class: urn:content-classes:message X-MS-Has-Attach: X-MimeOLE: Produced By Microsoft Exchange V6.5 X-MS-TNEF-Correlator: Thread-Topic: runit not collecting zombies thread-index: Ach3gFr9idfGeWyfQnGMsOOuWpf8FAASIYsg Xref: news.gmane.org gmane.comp.sysutils.supervision.general:1651 Archived-At: Actually a number of the updated packages *could* be the cause of this issue. The package that I would look at closest is the bash upgrade as runit scripts depend on the built-in exec command properly replacing the shell with the same process id. Next see if the util-linux package upgrade caused the problem. Last and most unlikely is the sysVinit upgrade. I would guess downgrading one package at a time and checking for zombies is the simplest methodology. I don't think the kernel is implicated if you haven't restarted the machine. Again if I was to point a finger it would be vaguely in the direction of the bash upgrade. As bashtastic as it is it is not un-heard of for bash to have some quirks version to version. R -----Original Message----- From: Alex Efros [mailto:powerman@powerman.name]=20 Sent: 25 February 2008 07:31 To: supervision@list.skarnet.org Subject: Re: runit not collecting zombies Hi! I've just got new information. One of our servers doesn't have that zombie issue. But it wasn't updated for about 6 months because it previous admin retired. Now I begin updating it. At first I updated simple tools and libraries, which shouldn't broke anything, and now I'm working on updating more critical things like toolchain, LAMP and few system tools like PAM and udev. Current uptime is 128 days. And know what? I have just noticed a lot of zombies there! Usual trick 'chmod 0 /etc/runit/stopit; kill -CONT 1' did the work, but...=20 I have neither updated runit (or other critical packages like toolchain) nor reboot system - this issue with zombies has arisen after update of some simple utils and libraries!!!!! WTF??? How this can be possible?! If runit doesn't reload dynamic libraries on the fly or execute external utils (which I believe is not the case), then how update of some libraries and tools may result in runit failure to gather zombies? Only possible "explanation" which come in my mind is some CPU/RAM usage pattern which is happens while compiling all these packages somehow affect runit-init. And this isn't just "high load", because this server most of time under high enough load, in all ways (CPU+RAM+HDD). Below is list of updated packages, I can provide also versions "from" is was updated, if somebody will need it. Sat Feb 23 17:01:46 2008 >>> sys-apps/sandbox-1.2.18.1-r2 Sat Feb 23 17:02:21 2008 >>> sys-apps/portage-2.1.4.4 Sat Feb 23 21:48:03 2008 >>> app-misc/colordiff-1.0.6-r1 Sun Feb 24 02:17:13 2008 >>> sys-libs/ncurses-5.6-r2 Sun Feb 24 02:17:57 2008 >>> app-arch/bzip2-1.0.4-r1 Sun Feb 24 02:18:16 2008 >>> sys-devel/gnuconfig-20070724 Sun Feb 24 02:21:45 2008 >>> media-libs/freetype-2.3.5-r2 Sun Feb 24 02:24:49 2008 >>> dev-util/pkgconfig-0.22 Sun Feb 24 02:26:01 2008 >>> sys-libs/cracklib-2.8.12 Sun Feb 24 02:26:33 2008 >>> app-misc/pax-utils-0.1.16 Sun Feb 24 02:26:59 2008 >>> sys-devel/gcc-config-1.4.0-r4 Sun Feb 24 02:27:49 2008 >>> sys-libs/timezone-data-2007k Sun Feb 24 02:29:33 2008 >>> media-libs/t1lib-5.0.2-r1 Sun Feb 24 02:32:08 2008 >>> dev-libs/libmcrypt-2.5.8 Sun Feb 24 02:35:28 2008 >>> dev-db/sqlite-3.5.3 Sun Feb 24 02:36:52 2008 >>> sys-fs/sysfsutils-2.1.0 Sun Feb 24 02:37:13 2008 >>> net-analyzer/netselect-0.3-r2 Sun Feb 24 02:37:30 2008 >>> sys-process/cronbase-0.3.2-r1 Sun Feb 24 02:40:02 2008 >>> dev-lang/spidermonkey-1.7.0 Sun Feb 24 02:42:02 2008 >>> app-arch/cpio-2.9-r1 Sun Feb 24 02:42:43 2008 >>> sys-process/acct-6.3.5-r2 Sun Feb 24 02:43:16 2008 >>> app-portage/portage-utils-0.1.29 Sun Feb 24 02:56:51 2008 >>> app-arch/p7zip-4.57 Sun Feb 24 02:58:10 2008 >>> sys-fs/reiserfsprogs-3.6.19-r2 Sun Feb 24 02:58:53 2008 >>> net-proxy/3proxy-0.5.3j Sun Feb 24 02:59:19 2008 >>> net-analyzer/traceroute-2.0.9-r1 Sun Feb 24 03:01:08 2008 >>> sys-kernel/linux-headers-2.6.23-r3 Sun Feb 24 03:04:43 2008 >>> app-shells/bash-3.2_p17-r1 Sun Feb 24 03:07:02 2008 >>> media-libs/libpng-1.2.24 Sun Feb 24 03:09:36 2008 >>> dev-libs/libpcre-7.6-r1 Sun Feb 24 03:13:22 2008 >>> media-libs/tiff-3.8.2-r3 Sun Feb 24 03:14:18 2008 >>> sys-apps/less-418 Sun Feb 24 03:24:46 2008 >>> app-portage/eix-0.10.2 Sun Feb 24 03:26:09 2008 >>> sys-apps/pciutils-2.2.9 Sun Feb 24 03:27:51 2008 >>> net-misc/netkit-telnetd-0.17-r8 Sun Feb 24 03:34:33 2008 >>> sys-libs/readline-5.2_p12-r1 Sun Feb 24 03:36:50 2008 >>> sys-apps/sysvinit-2.86-r10 Sun Feb 24 03:38:53 2008 >>> sys-apps/ed-0.8 Sun Feb 24 03:41:09 2008 >>> net-misc/iputils-20070202 Sun Feb 24 04:06:57 2008 >>> dev-libs/libxml2-2.6.30-r1 Sun Feb 24 04:53:51 2008 >>> sys-devel/gettext-0.17 Sun Feb 24 04:55:04 2008 >>> sys-apps/sed-4.1.5 Sun Feb 24 04:56:47 2008 >>> sys-devel/m4-1.4.10 Sun Feb 24 04:57:34 2008 >>> sys-apps/man-1.6f Sun Feb 24 04:58:44 2008 >>> sys-devel/flex-2.5.33-r3 Sun Feb 24 04:59:48 2008 >>> dev-libs/libgpg-error-1.6 Sun Feb 24 05:02:31 2008 >>> sys-apps/findutils-4.3.11 Sun Feb 24 05:04:23 2008 >>> sys-apps/gawk-3.1.5-r5 Sun Feb 24 05:05:38 2008 >>> app-editors/nano-2.0.7 Sun Feb 24 05:06:49 2008 >>> dev-util/dialog-1.1.20071028 Sun Feb 24 05:07:51 2008 >>> sys-apps/kbd-1.13-r1 Sun Feb 24 05:10:45 2008 >>> app-arch/tar-1.19-r1 Sun Feb 24 05:11:23 2008 >>> sys-apps/ucspi-tcp-0.88-r16 Sun Feb 24 05:12:49 2008 >>> sys-apps/man-pages-2.76 Sun Feb 24 05:13:23 2008 >>> sys-devel/bc-1.06-r6 Sun Feb 24 05:13:34 2008 >>> virtual/editor-0 Sun Feb 24 05:26:40 2008 >>> sys-devel/binutils-2.18-r1 Sun Feb 24 05:28:49 2008 >>> sys-libs/com_err-1.40.4 Sun Feb 24 05:41:56 2008 >>> sys-libs/db-4.5.20_p2 Sun Feb 24 05:43:16 2008 >>> sys-libs/ss-1.40.4 Sun Feb 24 05:43:38 2008 >>> sys-apps/paxctl-0.5 Sun Feb 24 05:46:54 2008 >>> sys-fs/e2fsprogs-1.40.4 Sun Feb 24 05:49:59 2008 >>> sys-apps/util-linux-2.13-r2 Sun Feb 24 05:54:34 2008 >>> sys-apps/parted-1.8.8 Sun Feb 24 05:56:03 2008 >>> net-mail/fetchmail-6.3.8-r1 Sun Feb 24 06:01:18 2008 >>> net-misc/ntp-4.2.4_p4 Sun Feb 24 06:15:26 2008 >>> dev-lang/perl-5.8.8-r4 Sun Feb 24 06:16:04 2008 >>> sys-devel/autoconf-2.61-r1 Sun Feb 24 06:16:20 2008 >>> app-admin/perl-cleaner-1.05 Sun Feb 24 06:18:05 2008 >>> dev-libs/libtasn1-1.2 Sun Feb 24 06:18:25 2008 >>> sys-apps/help2man-1.36.4 Sun Feb 24 06:20:09 2008 >>> net-misc/rsync-2.6.9-r5 Sun Feb 24 06:22:37 2008 >>> sys-devel/libtool-1.5.26 Sun Feb 24 07:56:47 2008 >>> sys-apps/busybox-1.8.2 Sun Feb 24 07:57:13 2008 >>> sys-apps/slocate-3.1-r1 Sun Feb 24 07:58:20 2008 >>> net-libs/libpcap-0.9.8 Sun Feb 24 08:02:15 2008 >>> dev-libs/libgcrypt-1.4.0-r1 Sun Feb 24 08:03:07 2008 >>> dev-perl/DBI-1.601 Sun Feb 24 08:03:47 2008 >>> x11-misc/read-edid-1.4.1-r1 Sun Feb 24 08:05:52 2008 >>> net-dns/libidn-1.0-r1 Sun Feb 24 08:07:13 2008 >>> sys-process/psmisc-22.6 Sun Feb 24 08:28:13 2008 >>> net-fs/samba-3.0.28 Sun Feb 24 08:30:00 2008 >>> dev-util/strace-4.5.16 Sun Feb 24 08:35:23 2008 >>> net-analyzer/rrdtool-1.2.23-r1 Sun Feb 24 08:38:41 2008 >>> dev-libs/libxslt-1.1.22 Sun Feb 24 08:40:32 2008 >>> app-crypt/opencdk-0.6.6 Sun Feb 24 08:41:20 2008 >>> net-analyzer/lft-3.0 Sun Feb 24 08:45:39 2008 >>> net-analyzer/nmap-4.53 Sun Feb 24 08:46:01 2008 >>> net-misc/whois-4.7.24 Sun Feb 24 08:47:51 2008 >>> net-analyzer/tcpdump-3.9.8 Sun Feb 24 08:57:19 2008 >>> net-libs/gnutls-2.0.4 Sun Feb 24 09:01:41 2008 >>> net-misc/curl-7.17.1 Sun Feb 24 09:08:55 2008 >>> dev-lang/python-2.4.4-r6 Sun Feb 24 09:10:28 2008 >>> sys-apps/file-4.23 Sun Feb 24 09:10:44 2008 >>> app-admin/eselect-vi-1.1.5 Sun Feb 24 09:10:55 2008 >>> app-admin/eselect-ctags-1.3 Sun Feb 24 09:11:55 2008 >>> dev-util/ctags-5.7 Sun Feb 24 09:12:25 2008 >>> sys-apps/debianutils-2.28.2 Sun Feb 24 09:12:45 2008 >>> app-portage/gentoolkit-0.2.3-r1 Sun Feb 24 09:13:18 2008 >>> sys-apps/baselayout-1.12.11.1 Sun Feb 24 09:14:03 2008 >>> sys-apps/module-init-tools-3.4 Sun Feb 24 09:30:04 2008 >>> sys-kernel/hardened-sources-2.6.23-r7 Sun Feb 24 09:31:40 2008 >>> net-dialup/ppp-2.4.4-r14 Sun Feb 24 09:33:01 2008 >>> sys-apps/lm_sensors-2.10.4 Sun Feb 24 09:34:21 2008 >>> net-firewall/iptables-1.3.8-r3 Sun Feb 24 09:36:33 2008 >>> app-editors/vim-core-7.1.123 Sun Feb 24 09:41:04 2008 >>> app-editors/vim-7.1.123 Sun Feb 24 10:08:14 2008 >>> dev-lang/php-5.2.4_pre200708051230-r2 Sun Feb 24 10:09:42 2008 >>> dev-php/PEAR-PEAR-1.6.2-r1 Sun Feb 24 10:09:53 2008 >>> app-admin/eselect-fontconfig-1.0 Sun Feb 24 10:13:11 2008 >>> media-libs/fontconfig-2.5.0-r1 Sun Feb 24 10:13:49 2008 >>> media-fonts/corefonts-1-r4 Sun Feb 24 10:35:59 2008 >>> media-gfx/imagemagick-6.3.7.9 Sun Feb 24 11:02:04 2008 >>> dev-libs/glib-2.14.6 Sun Feb 24 11:02:30 2008 >>> app-arch/rar-3.7.1 These packages was installed, not updated: Sun Feb 24 18:10:56 2008 >>> app-misc/mc-mp-4.1.40_pre9-r1 Sun Feb 24 23:47:46 2008 >>> app-forensics/chkrootkit-0.47 Sun Feb 24 23:48:45 2008 >>> sys-process/lsof-4.78-r1 Sun Feb 24 23:49:00 2008 >>> app-forensics/rkhunter-1.2.9 P.S. Gerrit: runit is really cool, but this bug (unfixed for about a year!) drives me crazy... :( --=20 WBR, Alex.