Re: hello - hanging services

[Jean-Michel Bruenn <jean.bruenn@ip-minds.de>]

> >> 
> >> Difficult to implement?
> > 
> > Yes.
> 
>  More precisely, it's not so much "difficult to implement" (I've done
> it for a paying customer's project) as "impossible to do without specific
> support in the service you're trying to manage".
>  In other words, what Jean-Michel wants is a software watchdog; it can
> be done, but it's pretty intrusive. It requires having a library, a daemon,
> and making library calls in the managed process' source, sending messages
> to the daemon by doing so. The daemon is configured with a certain policy
> that decides "the service is running fine" or "the service has hung"
> depending on the frequency of the messages it receives.
> 
>  It's doable, and a watchdog library/daemon may even have its place in
> a supervision suite (I'll think about it), but it certainly has nothing
> to do with purely external process management tools such as runsvdir/runsv
> or svscan/supervise. It's a whole piece of software on its own.
> 
>  I'm certain that a lot of open source software watchdogs already exist
> out there. I'm also certain that none of them is as lightweight and easy
> to use as I'd like, but that's another story.

In fact i was thinking about something more simple, i guess you guys
know nagios? similar to nagios - Just run a command, check for output
or timeout, for example for apache, you write a script called
"hangcheck" which gets run all X seconds by runit. This script
contains something like:

#!/bin/sh
service=apache2
exec curl 127.0.0.1 || sv restart $service

e.g:
wdp@localhost:~$ curl 127.0.0.1 || echo "didnt work"
curl: (7) couldn't connect to host
didnt work

So the idea is, you can define with which command to test a service
(you don't need to use this at all) and runit is just periodically
running the hangcheck script - the hangcheck script itself is just
running a command, and deciding be exit-code whether to do something or
not (so this can be used to mail someone about a hanging/not responding
service, or to restart this service.

So there's no need for any special scripting or any special algorithm.
And if i'm right there's not much work to be done in runit - just: if
[ -f hangcheck ]; then ./hangcheck; fi (of course with the periodic
timer set, let's say 10 seconds? 1 minute?)

Cheers