hello - hanging services

hello - hanging services

[Jean-Michel Bruenn]

Hey, 

i'm curious what happens with hung(hanging?) services (or zombies) is it
possible with runit to detect those and restart the service?

Cheers

Re: hello - hanging services

[Jean-Michel Bruenn]

Hello,

thanks for your answer. Wouldn't it be a good improvement for runit,
if it would take care of hanging tasks, also? There's "run", "finish"
and the "log" stuff - wouldn't it be possible to add "check" as script,
which is running a command all X seconds and if it gets a response it
knows "ah okay, the service is still running" and if it gets no
response "oh, the service seems to have died, let's restart it"?

Of course, totally optional, up to the user whether to use that or not.

Difficult to implement?

Cheers

On Tue, 17 Aug 2010
13:13:55 -0400 (EDT) Charlie Brady
<charlieb-supervise@budge.apana.org.au> wrote:

> 
> On Tue, 17 Aug 2010, Jean-Michel Bruenn wrote:
> 
> > Hey, 
> > 
> > i'm curious what happens with hung(hanging?) services (or zombies) is it
> > possible with runit to detect those and restart the service?
> 
> hung/hanging services and zombies are different things. A zombie is a 
> process which doesn't exist - it's just a process remnant - a status 
> report which the kernel is hanging onto waiting for someone to ask for it. 
> A hung/hanging services is a real process, but one which isn't doing what 
> you want it to do. runit does not detect and restart those. It only 
> restarts processes which have died.


-- 
Jean-Michel Bruenn <jean.bruenn@ip-minds.de>

Re: hello - hanging services

[Charlie Brady]

On Tue, 17 Aug 2010, Jean-Michel Bruenn wrote:

> Hello,
> 
> thanks for your answer. Wouldn't it be a good improvement for runit,
> if it would take care of hanging tasks, also?

You gotta detect them first, which is a non-trivial problem 
(algorithmically impossible in general - 
http://en.wikipedia.org/wiki/Halting_problem).

> There's "run", "finish"
> and the "log" stuff - wouldn't it be possible to add "check" as script,

check script already exists, but not what you are suggesting:

http://manpages.ubuntu.com/manpages/jaunty/man8/sv.8.html

> which is running a command all X seconds and if it gets a response it
> knows "ah okay, the service is still running" and if it gets no
> response "oh, the service seems to have died, let's restart it"?
> 
> Of course, totally optional, up to the user whether to use that or not.
> 
> Difficult to implement?

Yes.

Please check the archives - this has been discussed previously.

> Cheers
> 
> On Tue, 17 Aug 2010
> 13:13:55 -0400 (EDT) Charlie Brady
> <charlieb-supervise@budge.apana.org.au> wrote:
> 
> > 
> > On Tue, 17 Aug 2010, Jean-Michel Bruenn wrote:
> > 
> > > Hey, 
> > > 
> > > i'm curious what happens with hung(hanging?) services (or zombies) is it
> > > possible with runit to detect those and restart the service?
> > 
> > hung/hanging services and zombies are different things. A zombie is a 
> > process which doesn't exist - it's just a process remnant - a status 
> > report which the kernel is hanging onto waiting for someone to ask for it. 
> > A hung/hanging services is a real process, but one which isn't doing what 
> > you want it to do. runit does not detect and restart those. It only 
> > restarts processes which have died.
> 
> 
> 

Re: hello - hanging services

[Laurent Bercot]

>> which is running a command all X seconds and if it gets a response it
>> knows "ah okay, the service is still running" and if it gets no
>> response "oh, the service seems to have died, let's restart it"?
>> 
>> Difficult to implement?
> 
> Yes.

 More precisely, it's not so much "difficult to implement" (I've done
it for a paying customer's project) as "impossible to do without specific
support in the service you're trying to manage".
 In other words, what Jean-Michel wants is a software watchdog; it can
be done, but it's pretty intrusive. It requires having a library, a daemon,
and making library calls in the managed process' source, sending messages
to the daemon by doing so. The daemon is configured with a certain policy
that decides "the service is running fine" or "the service has hung"
depending on the frequency of the messages it receives.

 It's doable, and a watchdog library/daemon may even have its place in
a supervision suite (I'll think about it), but it certainly has nothing
to do with purely external process management tools such as runsvdir/runsv
or svscan/supervise. It's a whole piece of software on its own.

 I'm certain that a lot of open source software watchdogs already exist
out there. I'm also certain that none of them is as lightweight and easy
to use as I'd like, but that's another story.

-- 
 Laurent

Re: hello - hanging services

[Jean-Michel Bruenn]

> >> 
> >> Difficult to implement?
> > 
> > Yes.
> 
>  More precisely, it's not so much "difficult to implement" (I've done
> it for a paying customer's project) as "impossible to do without specific
> support in the service you're trying to manage".
>  In other words, what Jean-Michel wants is a software watchdog; it can
> be done, but it's pretty intrusive. It requires having a library, a daemon,
> and making library calls in the managed process' source, sending messages
> to the daemon by doing so. The daemon is configured with a certain policy
> that decides "the service is running fine" or "the service has hung"
> depending on the frequency of the messages it receives.
> 
>  It's doable, and a watchdog library/daemon may even have its place in
> a supervision suite (I'll think about it), but it certainly has nothing
> to do with purely external process management tools such as runsvdir/runsv
> or svscan/supervise. It's a whole piece of software on its own.
> 
>  I'm certain that a lot of open source software watchdogs already exist
> out there. I'm also certain that none of them is as lightweight and easy
> to use as I'd like, but that's another story.

In fact i was thinking about something more simple, i guess you guys
know nagios? similar to nagios - Just run a command, check for output
or timeout, for example for apache, you write a script called
"hangcheck" which gets run all X seconds by runit. This script
contains something like:

#!/bin/sh
service=apache2
exec curl 127.0.0.1 || sv restart $service

e.g:
wdp@localhost:~$ curl 127.0.0.1 || echo "didnt work"
curl: (7) couldn't connect to host
didnt work

So the idea is, you can define with which command to test a service
(you don't need to use this at all) and runit is just periodically
running the hangcheck script - the hangcheck script itself is just
running a command, and deciding be exit-code whether to do something or
not (so this can be used to mail someone about a hanging/not responding
service, or to restart this service.

So there's no need for any special scripting or any special algorithm.
And if i'm right there's not much work to be done in runit - just: if
[ -f hangcheck ]; then ./hangcheck; fi (of course with the periodic
timer set, let's say 10 seconds? 1 minute?)

Cheers

Re: hello - hanging services

[Charlie Brady]

On Wed, 18 Aug 2010, Jean-Michel Bruenn wrote:

> In fact i was thinking about something more simple, i guess you guys
> know nagios? similar to nagios - Just run a command, check for output
> or timeout, for example for apache, you write a script called
> "hangcheck" which gets run all X seconds by runit. This script
> contains something like:
> 
> #!/bin/sh
> service=apache2
> exec curl 127.0.0.1 || sv restart $service
> 
> e.g:
> wdp@localhost:~$ curl 127.0.0.1 || echo "didnt work"
> curl: (7) couldn't connect to host
> didnt work
> 
> So the idea is, you can define with which command to test a service
> (you don't need to use this at all) and runit is just periodically
> running the hangcheck script - the hangcheck script itself is just
> running a command, and deciding be exit-code whether to do something or
> not (so this can be used to mail someone about a hanging/not responding
> service, or to restart this service.
> 
> So there's no need for any special scripting or any special algorithm.
> And if i'm right there's not much work to be done in runit - just: if
> [ -f hangcheck ]; then ./hangcheck; fi (of course with the periodic
> timer set, let's say 10 seconds? 1 minute?)

There are many complications that you probably haven't thought of. What 
would runit do if the hangcheck script itself hangs? How might you 
compromise the existing functionality of runit by adding this feature? How 
many race conditions do you introduce?

IMO you don't need to have this functionality in runit, which is already 
doing it's specified task well. You want something else to act as a 
service watchdog. You use another tool to do that - for instance, nagios.

Re: hello - hanging services

[Jean-Michel Bruenn]

On Wed, 18 Aug 2010 11:23:41 -0400 (EDT)
Charlie Brady <charlieb-supervision@budge.apana.org.au> wrote:

> There are many complications that you probably haven't thought of. What 
> would runit do if the hangcheck script itself hangs? How might you 
> compromise the existing functionality of runit by adding this feature? How 
> many race conditions do you introduce?

I understand your thoughts about this, and yes i have thought about
this, too. But let's make it clear: This can happen with runit as it
is now, also: a weird written run-script or a broken log-script might
compromise the existing functionality of runit (if it doesnt, adding a
new variant like a hangcheck-script wouldn't do so neither). I mean:
what happens currently if one of the services which you're trying to
start hangs? I havent tried yet, so i guess only the service which
you're trying to start would be compromised - not whole runit. And this
wouldn't be the case with my suggestion neither.

Last but not Least: it's an optional feature, just like the log stuff -
You can use it, you don't need to. If some users feel like using it
they know about the risks, if there are really any (as explained above).

> IMO you don't need to have this functionality in runit, which is already 
> doing it's specified task well. You want something else to act as a 
> service watchdog. You use another tool to do that - for instance, nagios.

Probably you're right, though i don't exactly understand your
argumentation because: Runit is starting crashed processes (this
shouldn't be the job of an Init-System - the job of an init-system is
starting processes, not making sure that they're up and running -
thats the job of a software-watchdog). BUT: runit is doing exactly
this. Runit is taking care that your service is up and running, by
restarting it if its crashing - By argueing that "checking whether a
service is responding (and thus working) is not the job of runit", you
might also argue that "restarting a crashed job is not runit's job".

To point it out again: It's just "optionally" for users who'd like such
a feature, extending the current functionality of restarting a crashed
process - If you, or someone else, feels like "this could hurt my whole
system" you don't need to use it. This doesn't make such a feature
useless or bad.

And back to topic: i didn't wanted to request such a feature, i was
just asking whether it would be difficult to implement/possible.
Whether developers or users implement something like this, is not up to
me :)

P.S: i get your mails twice, purpose? :-)

Cheers

-- 
Jean-Michel Bruenn <jean.bruenn@ip-minds.de>

Re: hello - hanging services

[Laurent Bercot]

> I understand your thoughts about this, and yes i have thought about
> this, too. But let's make it clear: This can happen with runit as it
> is now, also: a weird written run-script or a broken log-script might
> compromise the existing functionality of runit (if it doesnt, adding a
> new variant like a hangcheck-script wouldn't do so neither). I mean:
> what happens currently if one of the services which you're trying to
> start hangs? I havent tried yet, so i guess only the service which
> you're trying to start would be compromised - not whole runit. And this
> wouldn't be the case with my suggestion neither.

 The problem is that your suggestion affects the reliability of the
service you want to check.
 If ./run hangs, well, the service hangs. ./run IS the service: of
course you need to write the script properly if you want the service
to function properly. There's nothing we can do about that. Same for
the logger, which is also a service (albeit a special one).
 If ./hangcheck hangs, then what should be the default policy? To be
congruent with a watchdog's purpose, you should restart the service.
But then, you might have a buggy ./hangcheck script and a perfectly
functional service, and restarting it for no good reason is a decrease
in service availability and reliability (and a waste of resources).
By adding ./hangcheck support, you are adding a dependency, and making
the service architecture more fragile. That's what Charlie meant
(I think).

 Small is beautiful for a reason: small has less hidden costs.
Everytime you want to add a feature, look for the hidden costs.
Sometimes the feature is worth paying them. Most of the time it's not.


> Probably you're right, though i don't exactly understand your
> argumentation because: Runit is starting crashed processes (this
> shouldn't be the job of an Init-System - the job of an init-system is
> starting processes, not making sure that they're up and running -
> thats the job of a software-watchdog).

 Please read the list archives; this has been discussed at length.
What it comes down to is the duties of process 1, and process 1 *has*
to restart processes (at least one), in order to keep the system in
a usable state no matter what happens, no matter what dies. A supervision
architecture such as runit is a natural consequence of properly
implementing process 1's duties.

 You are mixing two different notions of 'up and running'.
 What runit does (and what any init system *should* do) is make sure
that the *process* corresponding to a given service has been properly
forked and exec'ed. As long as the process is there, runit is happy.
It's an external process management tool.
 What a software watchdog does is make sure that said process actually
does what is expected of it, as opposed to i.e. hang or busyloop. This
is more complex, because it requires knowledge of what the service is
supposed to do, and it generally can't be done without access to the
service's source code.
 

> BUT: runit is doing exactly
> this. Runit is taking care that your service is up and running, by
> restarting it if its crashing - By argueing that "checking whether a
> service is responding (and thus working) is not the job of runit", you
> might also argue that "restarting a crashed job is not runit's job".

 No. runit's job is process management. runit is there to ensure that
the process tree you want is always there; and that includes restarting
crashed processes if needed. But runit's job is not making sure that
every process in the process tree is doing exactly what's it's supposed
to do. Again, there's a difference between "process A is up", which runit
can and should control, and "process A is behaving as expected", which
can only be controlled by some A-specific watchdog.

 And since this is Unix, two different things should be handled by two
different tools.

-- 
 Laurent

Re: hello - hanging services

[Nicolás de la Torre]

I must be on this list by mistake, please unsubscribe.

2010/8/19 Laurent Bercot <ska-supervision@skarnet.org>
>
> > I understand your thoughts about this, and yes i have thought about
> > this, too. But let's make it clear: This can happen with runit as it
> > is now, also: a weird written run-script or a broken log-script might
> > compromise the existing functionality of runit (if it doesnt, adding a
> > new variant like a hangcheck-script wouldn't do so neither). I mean:
> > what happens currently if one of the services which you're trying to
> > start hangs? I havent tried yet, so i guess only the service which
> > you're trying to start would be compromised - not whole runit. And this
> > wouldn't be the case with my suggestion neither.
>
>  The problem is that your suggestion affects the reliability of the
> service you want to check.
>  If ./run hangs, well, the service hangs. ./run IS the service: of
> course you need to write the script properly if you want the service
> to function properly. There's nothing we can do about that. Same for
> the logger, which is also a service (albeit a special one).
>  If ./hangcheck hangs, then what should be the default policy? To be
> congruent with a watchdog's purpose, you should restart the service.
> But then, you might have a buggy ./hangcheck script and a perfectly
> functional service, and restarting it for no good reason is a decrease
> in service availability and reliability (and a waste of resources).
> By adding ./hangcheck support, you are adding a dependency, and making
> the service architecture more fragile. That's what Charlie meant
> (I think).
>
>  Small is beautiful for a reason: small has less hidden costs.
> Everytime you want to add a feature, look for the hidden costs.
> Sometimes the feature is worth paying them. Most of the time it's not.
>
>
> > Probably you're right, though i don't exactly understand your
> > argumentation because: Runit is starting crashed processes (this
> > shouldn't be the job of an Init-System - the job of an init-system is
> > starting processes, not making sure that they're up and running -
> > thats the job of a software-watchdog).
>
>  Please read the list archives; this has been discussed at length.
> What it comes down to is the duties of process 1, and process 1 *has*
> to restart processes (at least one), in order to keep the system in
> a usable state no matter what happens, no matter what dies. A supervision
> architecture such as runit is a natural consequence of properly
> implementing process 1's duties.
>
>  You are mixing two different notions of 'up and running'.
>  What runit does (and what any init system *should* do) is make sure
> that the *process* corresponding to a given service has been properly
> forked and exec'ed. As long as the process is there, runit is happy.
> It's an external process management tool.
>  What a software watchdog does is make sure that said process actually
> does what is expected of it, as opposed to i.e. hang or busyloop. This
> is more complex, because it requires knowledge of what the service is
> supposed to do, and it generally can't be done without access to the
> service's source code.
>
>
> > BUT: runit is doing exactly
> > this. Runit is taking care that your service is up and running, by
> > restarting it if its crashing - By argueing that "checking whether a
> > service is responding (and thus working) is not the job of runit", you
> > might also argue that "restarting a crashed job is not runit's job".
>
>  No. runit's job is process management. runit is there to ensure that
> the process tree you want is always there; and that includes restarting
> crashed processes if needed. But runit's job is not making sure that
> every process in the process tree is doing exactly what's it's supposed
> to do. Again, there's a difference between "process A is up", which runit
> can and should control, and "process A is behaving as expected", which
> can only be controlled by some A-specific watchdog.
>
>  And since this is Unix, two different things should be handled by two
> different tools.
>
> --
>  Laurent

Re: hello - hanging services

[Tobia Conforto]

Nicolás de la Torre wrote:
> I must be on this list by mistake, please unsubscribe.

I had tried to subscribe to this list months ago, because I wanted to
discuss a feature with the developers of runit. But I never got any
emails from it or from the list administrator. I thought the list was
dead or not working. Now I'm getting random pieces of discussions.

Can someone please fix this list software?

Tobia

Re: hello - hanging services

[Charlie Brady]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 963 bytes --]


On Fri, 20 Aug 2010, Tobia Conforto wrote:

> Nicolás de la Torre wrote:
> > I must be on this list by mistake, please unsubscribe.
> 
> I had tried to subscribe to this list months ago, because I wanted to
> discuss a feature with the developers of runit. But I never got any
> emails from it or from the list administrator. I thought the list was
> dead or not working. Now I'm getting random pieces of discussions.
> 
> Can someone please fix this list software?

Please contact the correct party. The unsubscribe address and the contact 
address concerning operation of the list are both included with every 
message you receive from the list.

...
Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm
List-Post: <mailto:supervision@list.skarnet.org>
List-Help: <mailto:supervision-help@list.skarnet.org>
List-Unsubscribe: <mailto:supervision-unsubscribe@list.skarnet.org>
List-Subscribe: <mailto:supervision-subscribe@list.skarnet.org>
...

RE: hello - hanging services

[Rehan]

Hi

Not that I need it but I don't usually get the list links in the mails I receive.

Cheers
R

> -----Original Message-----
> From: Charlie Brady [mailto:charlieb-supervision@budge.apana.org.au]
> Sent: 20 August 2010 16:01
> To: Tobia Conforto
> Cc: supervision@list.skarnet.org
> Subject: Re: hello - hanging services
> 
> 
> On Fri, 20 Aug 2010, Tobia Conforto wrote:
> 
> > Nicolás de la Torre wrote:
> > > I must be on this list by mistake, please unsubscribe.
> >
> > I had tried to subscribe to this list months ago, because I wanted to
> > discuss a feature with the developers of runit. But I never got any
> > emails from it or from the list administrator. I thought the list was
> > dead or not working. Now I'm getting random pieces of discussions.
> >
> > Can someone please fix this list software?
> 
> Please contact the correct party. The unsubscribe address and the contact
> address concerning operation of the list are both included with every
> message you receive from the list.
> 
> ...
> Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm
> List-Post: <mailto:supervision@list.skarnet.org>
> List-Help: <mailto:supervision-help@list.skarnet.org>
> List-Unsubscribe: <mailto:supervision-unsubscribe@list.skarnet.org>
> List-Subscribe: <mailto:supervision-subscribe@list.skarnet.org>
> ...

RE: hello - hanging services

[Charlie Brady]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1500 bytes --]


On Fri, 20 Aug 2010, Rehan wrote:

> Not that I need it but I don't usually get the list links in the mails I receive.

I bet you do. View full mail headers and you will see them.

> 
> Cheers
> R
> 
> > -----Original Message-----
> > From: Charlie Brady [mailto:charlieb-supervision@budge.apana.org.au]
> > Sent: 20 August 2010 16:01
> > To: Tobia Conforto
> > Cc: supervision@list.skarnet.org
> > Subject: Re: hello - hanging services
> > 
> > 
> > On Fri, 20 Aug 2010, Tobia Conforto wrote:
> > 
> > > Nicolás de la Torre wrote:
> > > > I must be on this list by mistake, please unsubscribe.
> > >
> > > I had tried to subscribe to this list months ago, because I wanted to
> > > discuss a feature with the developers of runit. But I never got any
> > > emails from it or from the list administrator. I thought the list was
> > > dead or not working. Now I'm getting random pieces of discussions.
> > >
> > > Can someone please fix this list software?
> > 
> > Please contact the correct party. The unsubscribe address and the contact
> > address concerning operation of the list are both included with every
> > message you receive from the list.
> > 
> > ...
> > Mailing-List: contact supervision-help@list.skarnet.org; run by ezmlm
> > List-Post: <mailto:supervision@list.skarnet.org>
> > List-Help: <mailto:supervision-help@list.skarnet.org>
> > List-Unsubscribe: <mailto:supervision-unsubscribe@list.skarnet.org>
> > List-Subscribe: <mailto:supervision-subscribe@list.skarnet.org>
> > ...
> 

Re: hello - hanging services

[Laurent Bercot]

> I had tried to subscribe to this list months ago, because I wanted to
> discuss a feature with the developers of runit. But I never got any
> emails from it or from the list administrator.

 You should have received an automated confirmation e-mail when your
subscription was confirmed. If you haven't, some misconfigured
antispam program probably stopped it. I'm sorry to hear that.

 You did not receive e-mails from the list in the few past months
because the list was simply inactive. It happens.
 

> I thought the list was dead or not working. Now I'm getting random
> pieces of discussions.

 The list is working fine. You should be getting whole discussions.
If you are not, please check your mail filtering software.

-- 
 Laurent