runit-1.8.0 available

runit-1.8.0 available

[Gerrit Pape]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

Hi, a new runit package, version 1.8.0, is available for testing

 http://smarden.org/runit/

The runit(8) program, the process no 1, has been fixed to reap dead
processes that re-parented to process no 1 (zombies) more thoroughly.
Instructions on how to use runit with upstart as init scheme have been
added, svlogd(8) has been changed to use a new source port for each log
message sent through udp, and this release includes a build fix for AIX.

If you use runit regularly, please contribute[0] to the project.

Regards, Gerrit.

[0] http://smarden.org/pape/#contribution

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: runit-1.8.0 available

[Alex Efros]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

Hi!

On Fri, Sep 21, 2007 at 11:12:48AM +0000, Gerrit Pape wrote:
> Instructions on how to use runit with upstart as init scheme have been

---cut---
Step 6: Replace /sbin/init
Now it is time to replace the sysvinit /sbin/init binary:
 # mv /sbin/init /sbin/init.sysv
 # ln -s runit-init /sbin/init
---cut---

IMO this is bad idea. I've used this few years, and then switch back to
using kernel param init= instead. This is because your linux distribution
from time to time may wish to update sysvinit package, and so it will
overwrite /sbin/init. And next reboot will be "surprise!", unless you
really-really careful and detect sysvinit upgrade and replace /sbin/init
with runit-init again after upgrade but before reboot. I've no idea how
often sysvinit package upgraded in other linux distributions, but in
Gentoo it upgrade, or at least recompile/reinstall every few months.

-- 
			WBR, Alex.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Fri, Sep 21, 2007 at 11:12:48AM +0000, Gerrit Pape wrote:
> Hi, a new runit package, version 1.8.0, is available for testing
> 
>  http://smarden.org/runit/
> 
> The runit(8) program, the process no 1, has been fixed to reap dead
> processes that re-parented to process no 1 (zombies) more thoroughly.

:-(

home ~ # uptime; ps ax | grep Z | wc
 17:35:01 up 11:37, 21 users,  load average: 0.00, 0.02, 0.00
   2024   12151   91273
home ~ # chmod -x /etc/runit/stopit 
home ~ # kill -CONT 1
home ~ # uptime; ps ax | grep Z | wc
 17:35:16 up 11:37, 21 users,  load average: 0.00, 0.01, 0.00
      2      19     134

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Gerrit Pape]

On Sat, Sep 22, 2007 at 05:37:24PM +0300, Alex Efros wrote:
> home ~ # uptime; ps ax | grep Z | wc
>  17:35:01 up 11:37, 21 users,  load average: 0.00, 0.02, 0.00
>    2024   12151   91273
> home ~ # chmod -x /etc/runit/stopit 
> home ~ # kill -CONT 1
> home ~ # uptime; ps ax | grep Z | wc
>  17:35:16 up 11:37, 21 users,  load average: 0.00, 0.01, 0.00
>       2      19     134

Hmm, what does
 # strings /proc/1/exe |grep Id
say?

Regards, Gerrit.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Mon, Sep 24, 2007 at 10:19:04AM +0000, Gerrit Pape wrote:
> On Sat, Sep 22, 2007 at 05:37:24PM +0300, Alex Efros wrote:
> > home ~ # uptime; ps ax | grep Z | wc
> >  17:35:01 up 11:37, 21 users,  load average: 0.00, 0.02, 0.00
> >    2024   12151   91273
> > home ~ # chmod -x /etc/runit/stopit 
> > home ~ # kill -CONT 1
> > home ~ # uptime; ps ax | grep Z | wc
> >  17:35:16 up 11:37, 21 users,  load average: 0.00, 0.01, 0.00
> >       2      19     134
> 
> Hmm, what does
>  # strings /proc/1/exe |grep Id
> say?

home ~ # strings /proc/1/exe |grep Id
$Id: 25da3b86f7bed4038b8a039d2f8e8c9bbcf0822b $

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Mon, Sep 24, 2007 at 10:19:04AM +0000, Gerrit Pape wrote:
> Hmm, what does
>  # strings /proc/1/exe |grep Id
> say?

For now - all servers ok, uptime 4 days. Maybe that was my mistake, and
zombies was found on server which wasn't rebooted with new runit yet...
but I've double-checked this before writing maillist, cos this is obvious.
Let's see how it will be going, 4 days is not enough for this issue.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Wed, Sep 26, 2007 at 04:46:23PM +0300, Alex Efros wrote:
> Let's see how it will be going, 4 days is not enough for this issue.

I've just checked servers. On _ALL_ servers I've unreaped zombies now.
Previous check was 14 hours ago, everything was clean.
Uptime on all servers is 6 days 23 hours.
On my home workstation 8 hours ago everything was clean, but now I've
zombies here too. Workstation uptime is 7 days 9 hours.

Amount of zombies on different servers between 15 and 9000, most have
300-600 zombies.

So, this runit version doesn't fixed zombie issue. :(

'chmod -x /etc/runit/stopit ; kill -CONT 1' trick works ok, so now at
least I don't have to reboot servers because of this issue.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Wed, Sep 26, 2007 at 04:46:23PM +0300, Alex Efros wrote:
> Let's see how it will be going, 4 days is not enough for this issue.

I've just checked servers. On _ALL_ servers I've unreaped zombies now.
Previous check was 14 hours ago, everything was clean.
Uptime on all servers is 6 days 23 hours.
On my home workstation 8 hours ago everything was clean, but now I've
zombies here too. Workstation uptime is 7 days 9 hours.

Amount of zombies on different servers between 15 and 9000, most have
300-600 zombies.

So, this runit version doesn't fixed zombie issue. :(

'chmod -x /etc/runit/stopit ; kill -CONT 1' trick works ok, so now at
least I don't have to reboot servers because of this issue.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

Sorry for duplicating messages - I was changing my email, subscribed from
new address, and send message from both new and old addresses.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Sat, Sep 29, 2007 at 04:03:51PM +0300, Alex Efros wrote:
> I've just checked servers. On _ALL_ servers I've unreaped zombies now.
> Previous check was 14 hours ago, everything was clean.
> Uptime on all servers is 6 days 23 hours.

Now it happens again. Uptime on all servers is 13 days 16 hours.
Previous check was 2 days ago.

Looks like this issue happens every ~6-6.5 days. Let's see when it happens
again...

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Gerrit Pape]

On Sat, Oct 06, 2007 at 08:49:23AM +0300, Alex Efros wrote:
> On Sat, Sep 29, 2007 at 04:03:51PM +0300, Alex Efros wrote:
> > I've just checked servers. On _ALL_ servers I've unreaped zombies now.
> > Previous check was 14 hours ago, everything was clean.
> > Uptime on all servers is 6 days 23 hours.
> 
> Now it happens again. Uptime on all servers is 13 days 16 hours.
> Previous check was 2 days ago.

Hmm, I don't know what's wrong.  'kill -CONT 1' makes the zombies go
away?  Does 'kill -CHLD 1' also work, or
'chmod -x /etc/runit/ctrlaltdel; kill -INT 1; chmod +x /etc/runit/ctrlaltdel'?

Regards, Gerrit.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Thu, Oct 11, 2007 at 12:53:49PM +0000, Gerrit Pape wrote:
> Hmm, I don't know what's wrong.  'kill -CONT 1' makes the zombies go
> away?  Does 'kill -CHLD 1' also work, or
> 'chmod -x /etc/runit/ctrlaltdel; kill -INT 1; chmod +x /etc/runit/ctrlaltdel'?

No, these commands doesn't fix this issue. Moreover, old trick with CONT
also don't work - or, more correctly, it MAY work, but it also REBOOT
server now, which shouldn't happen AFAIK! :(

# uptime; ps ax | grep Z | wc
 17:24:26 up 21 days,  3:43,  3 users,  load average: 0.00, 0.00, 0.06
    378    2269   16681
# kill -CHLD 1
# uptime; ps ax | grep Z | wc
 17:25:27 up 21 days,  3:44,  4 users,  load average: 0.00, 0.00, 0.05
    378    2269   16681
# chmod -x /etc/runit/ctrlaltdel
# kill -INT 1
# chmod +x /etc/runit/ctrlaltdel
# uptime; ps ax | grep Z | wc
 17:26:00 up 21 days,  3:45,  4 users,  load average: 0.00, 0.00, 0.05
    378    2269   16681
# chmod -x /etc/runit/stopit
# kill -CONT 1
# 
Broadcast message from root (Sat Oct 13 17:26:20 2007):

System is going down...

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Sat, Oct 13, 2007 at 08:30:17PM +0300, Alex Efros wrote:
> Moreover, old trick with CONT also don't work - or, more correctly, it
> MAY work, but it also REBOOT server now, which shouldn't happen AFAIK!

Now it happens again, but this time kill -CONT 1 works as expected and
doesn't reboot the servers. I've no idea why it reboot servers at previous time.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Sat, Oct 06, 2007 at 08:49:23AM +0300, Alex Efros wrote:
> > I've just checked servers. On _ALL_ servers I've unreaped zombies now.
> > Previous check was 14 hours ago, everything was clean.
> > Uptime on all servers is 6 days 23 hours.
> 
> Now it happens again. Uptime on all servers is 13 days 16 hours.
> Previous check was 2 days ago.
> 
> Looks like this issue happens every ~6-6.5 days. Let's see when it happens
> again...

It's funny, but I again got this issue simultaneously on all servers,
including workstation with uptime 2 days!

Most servers uptime is 21 day, but one server uptime is 5 days.

Maybe it doesn't depend on some time period, but instead depend on some time?

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[George Georgalis]

On Sun, Oct 14, 2007 at 12:27:54AM +0300, Alex Efros wrote:
>On Sat, Oct 06, 2007 at 08:49:23AM +0300, Alex Efros wrote:
>> > I've just checked servers. On _ALL_ servers I've unreaped zombies now.
>> > Previous check was 14 hours ago, everything was clean.
>> > Uptime on all servers is 6 days 23 hours.
>> 
>> Now it happens again. Uptime on all servers is 13 days 16 hours.
>> Previous check was 2 days ago.
>> 
>> Looks like this issue happens every ~6-6.5 days. Let's see when it happens
>> again...
>
>It's funny, but I again got this issue simultaneously on all servers,
>including workstation with uptime 2 days!
>
>Most servers uptime is 21 day, but one server uptime is 5 days.
>
>Maybe it doesn't depend on some time period, but instead depend on some time?

This thread(s) is so long it's become difficult to follow. Maybe
you could consolidate the important details into a summary. What
is the simplest way to reproduce the problem. What has been tried?
What factors are determined not related. What hypothesis, if any,
for resolution?

// George

-- 
George Georgalis, information system scientist <IXOYE><

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Mon, Oct 15, 2007 at 11:38:18PM -0400, George Georgalis wrote:
> This thread(s) is so long it's become difficult to follow. Maybe
> you could consolidate the important details into a summary. What
> is the simplest way to reproduce the problem. What has been tried?
> What factors are determined not related. What hypothesis, if any,
> for resolution?

Only known to me way to reproduce the problem - install new Gentoo server
and wait for about a week to see sshd zombies (as result of ssh-worms
trying to bruteforce ssh from time to time).

Tried? I tried to switch from runit-init to sysvinit, and this solved issue.
Also Gerrit suggested a workaround: running 'chmod -x /etc/runit/stopit;
kill -CONT 1' on system with unreaped zombies result in two things: first
all zombies are reaped, and second runit start reaping zombies again...
but after several days it stop reaping zombies again and we need to
chmod/kill again.

Not related... there several factors determined not related (like
grsecurity kernel patches), but that was while I wasn't sure this is bug
in runit.

Mostly strange thing is this happens as least for two people, at same time
after Gentoo upgrade. And that upgrade doesn't touch runit or toolchain -
nothing in this upgrade seems suspicious.

Only hypothesis I've - this issue related to date/time: it usually happens
at same time on all my servers (and looks like this related to global
date/time, and not to server uptime), and it usually repeats every 5-7 days.

I think easies way to solve this issue - if Gerrit provide test/debug
version of runit to me, which for example output it state/actions into log
file, and then he'll analyse that log file to find out what is going wrong.
Because looks like he unable to find this bug by just looking at the code.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[George Georgalis]

On Sat, Oct 20, 2007 at 10:59:50PM +0300, Alex Efros wrote:
>Hi!
>
>On Mon, Oct 15, 2007 at 11:38:18PM -0400, George Georgalis wrote:
>> This thread(s) is so long it's become difficult to follow. Maybe
>> you could consolidate the important details into a summary. What
>> is the simplest way to reproduce the problem. What has been tried?
>> What factors are determined not related. What hypothesis, if any,
>> for resolution?
>
>Only known to me way to reproduce the problem - install new Gentoo server
>and wait for about a week to see sshd zombies (as result of ssh-worms
>trying to bruteforce ssh from time to time).
>
>Tried? I tried to switch from runit-init to sysvinit, and this solved issue.
>Also Gerrit suggested a workaround: running 'chmod -x /etc/runit/stopit;
>kill -CONT 1' on system with unreaped zombies result in two things: first
>all zombies are reaped, and second runit start reaping zombies again...
>but after several days it stop reaping zombies again and we need to
>chmod/kill again.
>
>Not related... there several factors determined not related (like
>grsecurity kernel patches), but that was while I wasn't sure this is bug
>in runit.
>
>Mostly strange thing is this happens as least for two people, at same time
>after Gentoo upgrade. And that upgrade doesn't touch runit or toolchain -
>nothing in this upgrade seems suspicious.
>
>Only hypothesis I've - this issue related to date/time: it usually happens
>at same time on all my servers (and looks like this related to global
>date/time, and not to server uptime), and it usually repeats every 5-7 days.
>
>I think easies way to solve this issue - if Gerrit provide test/debug
>version of runit to me, which for example output it state/actions into log
>file, and then he'll analyse that log file to find out what is going wrong.
>Because looks like he unable to find this bug by just looking at the code.

it sounds like a signal is not reaching init, SIGPIPE?

The following sed to default sshd_config
	s/.*PasswordAuthentication.*/PasswordAuthentication no/
	s/.*UsePam.*/UsePam no/
will really cut back the impact of bad internet on public sshd port,
of course you will only be able to use keys (PKI/RSA) to connect.

// George

-- 
George Georgalis, information system scientist <IXOYE><

Re: runit-1.8.0 available

[Alex Efros]

Hi!

On Sat, Oct 20, 2007 at 06:11:26PM -0400, George Georgalis wrote:
> it sounds like a signal is not reaching init, SIGPIPE?

PIPE? You mean CHLD?

> The following sed to default sshd_config
> 	s/.*PasswordAuthentication.*/PasswordAuthentication no/
> 	s/.*UsePam.*/UsePam no/
> will really cut back the impact of bad internet on public sshd port,
> of course you will only be able to use keys (PKI/RSA) to connect.

Yeah, this is my default ssh configuration. :) But ssh worms anyway try to
connect (they doesn't know is senseless :)) and so ssh fork new processes
for these connections and these processes become unreaped zombies at some
point.

-- 
			WBR, Alex.

Re: runit-1.8.0 available

[Vincent Danen]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

* Alex Efros <powerman@powerman.name> [2007-10-21 01:20:51 +0300]:

>Hi!
>
>On Sat, Oct 20, 2007 at 06:11:26PM -0400, George Georgalis wrote:
>> it sounds like a signal is not reaching init, SIGPIPE?
>
>PIPE? You mean CHLD?
>
>> The following sed to default sshd_config
>> 	s/.*PasswordAuthentication.*/PasswordAuthentication no/
>> 	s/.*UsePam.*/UsePam no/
>> will really cut back the impact of bad internet on public sshd port,
>> of course you will only be able to use keys (PKI/RSA) to connect.
>
>Yeah, this is my default ssh configuration. :) But ssh worms anyway try to
>connect (they doesn't know is senseless :)) and so ssh fork new processes
>for these connections and these processes become unreaped zombies at some
>point.

The simplest thing to do would be to either use tcp_wrappers or run sshd
out of xinetd or (my favourite) ipsvd.  With some appropriate ACLs and
IP-based restrictions, you can reduce those attacks to *0* reaching sshd
as xinetd/ipsvd would deny them before even starting sshd.

-- 
Vincent Danen @ http://linsec.ca/

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: runit-1.8.0 available

[George Georgalis]

On Sun, Oct 21, 2007 at 01:20:51AM +0300, Alex Efros wrote:
>Hi!
>
>On Sat, Oct 20, 2007 at 06:11:26PM -0400, George Georgalis wrote:
>> it sounds like a signal is not reaching init, SIGPIPE?
>
>PIPE? You mean CHLD?

well I _meant_ PIPE but looking at signal(7)... it doesn't mean
exactly what I thought. I guess CHLD, I was thinking the case
where sshd doesn't handle PIPE properly (lame brute force tcp);
maybe SIGCHLD is what init is not getting, but should... I've not
given much thought to whether init should get the 'what' or 'why'
signal, is that established?


>> The following sed to default sshd_config
>> 	s/.*PasswordAuthentication.*/PasswordAuthentication no/
>> 	s/.*UsePam.*/UsePam no/
>> will really cut back the impact of bad internet on public sshd port,
>> of course you will only be able to use keys (PKI/RSA) to connect.
>
>Yeah, this is my default ssh configuration. :) But ssh worms anyway try to
>connect (they doesn't know is senseless :)) and so ssh fork new processes
>for these connections and these processes become unreaped zombies at some
>point.

I've never put sshd in supervise, nor noticed the (connection)
problem you describe. may the internal sshd spawning manages this
better than when run in foreground for ipsvd? Are you invoking
sshd in inetd style with ipsvd or exec sshd?

// George


-- 
George Georgalis, information system scientist <IXOYE><