Re: hello - hanging services

[Charlie Brady <charlieb-supervision@budge.apana.org.au>]

On Wed, 18 Aug 2010, Jean-Michel Bruenn wrote:

> In fact i was thinking about something more simple, i guess you guys
> know nagios? similar to nagios - Just run a command, check for output
> or timeout, for example for apache, you write a script called
> "hangcheck" which gets run all X seconds by runit. This script
> contains something like:
> 
> #!/bin/sh
> service=apache2
> exec curl 127.0.0.1 || sv restart $service
> 
> e.g:
> wdp@localhost:~$ curl 127.0.0.1 || echo "didnt work"
> curl: (7) couldn't connect to host
> didnt work
> 
> So the idea is, you can define with which command to test a service
> (you don't need to use this at all) and runit is just periodically
> running the hangcheck script - the hangcheck script itself is just
> running a command, and deciding be exit-code whether to do something or
> not (so this can be used to mail someone about a hanging/not responding
> service, or to restart this service.
> 
> So there's no need for any special scripting or any special algorithm.
> And if i'm right there's not much work to be done in runit - just: if
> [ -f hangcheck ]; then ./hangcheck; fi (of course with the periodic
> timer set, let's say 10 seconds? 1 minute?)

There are many complications that you probably haven't thought of. What 
would runit do if the hangcheck script itself hangs? How might you 
compromise the existing functionality of runit by adding this feature? How 
many race conditions do you introduce?

IMO you don't need to have this functionality in runit, which is already 
doing it's specified task well. You want something else to act as a 
service watchdog. You use another tool to do that - for instance, nagios.