From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.io/gmane.comp.sysutils.supervision.general/1513 Path: news.gmane.org!not-for-mail From: Alex Efros Newsgroups: gmane.comp.sysutils.supervision.general Subject: Re: runit not collecting zombies Date: Wed, 12 Sep 2007 23:28:42 +0300 Organization: asdfGroup Inc., http://powerman.asdfGroup.com/ Message-ID: <20070912202842.GJ12043@home.power> References: <35517.::ffff:77.75.72.5.1189613042.squirrel@mail.podgorny.cz> <20070912170450.GE12043@home.power> <200709121338.54750.mike@geekgene.com> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1189628928 8547 80.91.229.12 (12 Sep 2007 20:28:48 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Wed, 12 Sep 2007 20:28:48 +0000 (UTC) To: supervision@list.skarnet.org Original-X-From: supervision-return-1748-gcsg-supervision=m.gmane.org@list.skarnet.org Wed Sep 12 22:28:44 2007 Return-path: Envelope-to: gcsg-supervision@gmane.org Original-Received: from antah.skarnet.org ([212.85.147.14]) by lo.gmane.org with smtp (Exim 4.50) id 1IVYpI-0006AI-3g for gcsg-supervision@gmane.org; Wed, 12 Sep 2007 22:28:44 +0200 Original-Received: (qmail 26925 invoked by uid 76); 12 Sep 2007 20:29:05 -0000 Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Archive: Original-Received: (qmail 26918 invoked from network); 12 Sep 2007 20:29:05 -0000 Mail-Followup-To: supervision@list.skarnet.org Content-Disposition: inline In-Reply-To: <200709121338.54750.mike@geekgene.com> User-Agent: Mutt/1.5.16 (2007-06-09) Xref: news.gmane.org gmane.comp.sysutils.supervision.general:1513 Archived-At: Hi! On Wed, Sep 12, 2007 at 01:38:54PM -0600, Mike Buland wrote: > I'm just curious, but doesn't it sound like this is the first place to look > for the trouble? Unfortunately, as you point out, there are two differences > between the two systems, the one that works isn't using two of the hardened > patches, and is using a newer gcc. Have you reported these facts to the Yep. I dislike both idea to use non-hardened gcc on production servers and even more dislike idea to upgrade to gcc-4.1.1 without ability to safely disgrade after testing. Remember, this issue happens every ~week, so I should wait at least 3 weeks because saying 'huh, changing gcc solved issue'. I've tried to analyze this from other side. As I noted here: http://bugs.gentoo.org/show_bug.cgi?id=190261#c1 this issue happens at some time, and then repeated every 2-10 days. So, looks like something was changed on all my servers 2-10 days BEFORE this issue happens for the first time. Only changed thing was usual upgrade for some Gentoo packages. And I know when this issue happens for me first time: 2007-05-26. And I've logs for all package upgrades and server reboots for that period: Fri Apr 21 19:18:39 2006 >>> sys-process/runit-1.5.0 ... Kernel 2.6.16-hardened-r11 was used from Sep 10 12:46:47 GMT 2006 ... Sun Sep 10 17:42:41 2006 >>> sys-devel/gcc-3.4.6-r1 ... Mon Dec 18 02:25:38 2006 >>> sys-libs/glibc-2.3.6-r5 ... reboot (2.6.16-hardened-r11) at Sat Dec 23 23:58:49 GMT 2006 ... Mon Jan 1 21:35:05 2007 >>> sys-devel/gcc-3.4.6-r2 ... Sat Mar 31 01:45:24 2007 >>> sys-devel/gcc-3.4.6-r2 ... Sun Apr 1 13:37:43 2007 >>> dev-lang/perl-5.8.8-r2 Sun Apr 1 13:41:18 2007 >>> dev-lang/perl-5.8.8-r2 Sun Apr 1 13:41:49 2007 >>> dev-perl/Net-Daemon-0.39 Sun Apr 1 13:41:54 2007 >>> dev-perl/PlRPC-0.2018 Sun Apr 1 13:42:09 2007 >>> dev-perl/DBI-1.53 Sun Apr 1 13:42:26 2007 >>> dev-perl/DBD-mysql-3.0008 Sun Apr 1 17:59:45 2007 >>> app-misc/mime-types-7 Sun Apr 1 18:00:57 2007 >>> sys-apps/man-1.6e-r1 Sun Apr 1 18:07:55 2007 >>> sys-libs/db-4.3.29-r2 Sun Apr 1 18:08:07 2007 >>> app-portage/gentoolkit-0.2.3-r1 Sun Apr 8 18:12:28 2007 >>> sys-libs/ncurses-5.6 Sun Apr 8 18:13:15 2007 >>> sys-apps/file-4.20-r1 Wed Apr 11 03:08:33 2007 >>> sys-apps/man-pages-2.44 reboot (2.6.16-hardened-r11) at Fri Apr 27 21:55:13 GMT 2007 Sun May 6 19:05:48 2007 >>> sys-apps/debianutils-2.17.5 Sun May 6 19:08:07 2007 >>> dev-libs/apr-0.9.12 Sun May 6 19:11:34 2007 >>> dev-util/pkgconfig-0.21-r1 Sun May 6 19:11:54 2007 >>> sys-libs/timezone-data-2007d Sun May 6 19:12:48 2007 >>> dev-lang/spidermonkey-1.5-r2 Sun May 6 19:13:17 2007 >>> sys-devel/patch-2.5.9-r1 Sun May 6 19:13:24 2007 >>> sys-apps/hdparm-6.9-r1 Sun May 6 19:14:28 2007 >>> net-misc/rsync-2.6.9-r2 Sun May 6 19:15:34 2007 >>> dev-libs/pth-2.0.6 Sun May 6 19:15:37 2007 >>> sys-devel/binutils-config-1.9-r4 Sun May 6 19:19:57 2007 >>> app-shells/bash-3.2_p15-r1 Sun May 6 19:20:31 2007 >>> dev-util/dialog-1.1.20070227 Sun May 6 19:20:59 2007 >>> sys-apps/man-1.6e-r3 Sun May 6 19:22:14 2007 >>> media-libs/libpng-1.2.16 Sun May 6 19:23:48 2007 >>> media-libs/freetype-2.1.10-r3 Sun May 6 19:23:58 2007 >>> app-misc/ca-certificates-20070303-r1 Sun May 6 19:26:04 2007 >>> sys-libs/readline-5.2_p2 Sun May 6 19:27:49 2007 >>> dev-libs/libgpg-error-1.5 Sun May 6 19:28:43 2007 >>> sys-devel/m4-1.4.9 Sun May 6 19:30:40 2007 >>> sys-fs/e2fsprogs-1.39-r2 Sun May 6 19:31:19 2007 >>> app-editors/nano-2.0.4 Sun May 6 19:32:19 2007 >>> net-mail/fetchmail-6.3.8 Sun May 6 19:32:55 2007 >>> sys-devel/flex-2.5.33-r2 Sun May 6 19:33:25 2007 >>> sys-apps/baselayout-1.12.9-r2 Sun May 6 19:36:02 2007 >>> sys-apps/util-linux-2.12r-r6 Sun May 6 19:37:14 2007 >>> app-editors/vim-core-7.0.235 Sun May 6 19:38:27 2007 >>> dev-libs/libksba-1.0.0 Sun May 6 19:40:22 2007 >>> dev-libs/libxslt-1.1.20 Sun May 6 19:41:04 2007 >>> sys-apps/module-init-tools-3.2.2-r3 Sun May 6 19:47:27 2007 >>> app-editors/vim-7.0.235 Sun May 6 19:53:14 2007 >>> sys-kernel/hardened-sources-2.6.20-r2 Sun May 6 19:56:14 2007 >>> net-misc/curl-7.15.1-r1 Sun May 6 20:36:14 2007 >>> dev-db/mysql-5.0.38 Sun May 6 20:53:01 2007 >>> media-gfx/imagemagick-6.3.3 Sun May 6 20:53:19 2007 >>> sys-devel/gcc-config-1.3.16 Sun May 6 21:11:35 2007 >>> sys-libs/libstdc++-v3-3.3.6 Wed May 9 15:53:39 2007 >>> media-libs/freetype-2.3.3 Wed May 9 15:59:54 2007 >>> dev-lang/python-2.4.4 Wed May 9 16:04:33 2007 >>> mail-mta/netqmail-1.05-r8 Wed May 23 13:51:19 2007 >>> sys-apps/portage-2.1.2.7 Wed May 23 13:51:45 2007 >>> sys-libs/timezone-data-2007e Wed May 23 13:51:55 2007 >>> app-forensics/chkrootkit-0.47 Wed May 23 13:52:28 2007 >>> sys-libs/zlib-1.2.3-r1 Wed May 23 13:53:50 2007 >>> media-libs/libpng-1.2.18 Wed May 23 13:56:07 2007 >>> media-libs/freetype-2.3.4-r2 Wed May 23 14:44:45 2007 >>> dev-db/mysql-5.0.40 Wed May 23 14:50:22 2007 >>> dev-lang/python-2.4.4-r4 Wed May 23 14:50:24 2007 >>> app-admin/python-updater-0.2 Wed May 23 14:51:36 2007 >>> sys-apps/util-linux-2.12r-r7 Wed May 23 14:52:04 2007 >>> sys-apps/gradm-2.1.10.200702231759 Wed May 23 14:59:26 2007 >>> app-crypt/gnupg-1.4.7-r1 reboot (2.6.16-hardened-r11) at Sat May 26 10:37:41 GMT 2007 reboot (2.6.16-hardened-r11) at Sat Jun 2 14:54:20 GMT 2007 Sun Jun 3 13:04:10 2007 >>> app-portage/eix-0.9.1 reboot (2.6.16-hardened-r11) at Sat Jun 9 14:56:38 GMT 2007 Mon Jun 11 13:00:46 2007 >>> sys-process/runit-1.7.2 reboot (2.6.16-hardened-r11) at Mon Jun 11 13:06:48 GMT 2007 reboot (2.6.16-hardened-r11) at Sat Jun 16 03:59:03 GMT 2007 reboot (2.6.20-hardened-r2) at Sat Jun 16 04:26:30 GMT 2007 ... Thu Jun 14 23:12:38 2007 >>> sys-libs/glibc-2.5-r3 Neither runit, nor gcc, glibc or kernel was upgraded in May (kernel sources was unpacked in /usr/src at May 6, but it was compiled and boot only Jun 16). Any ideas how upgrading THESE packages may affect zombie reaping in already running runit? :-/ -- WBR, Alex.