Re: Respawn limit for runsv?

[Charlie Brady <charlieb-supervision@budge.apana.org.au>]

On Sat, 12 Feb 2005, Lars Kellogg-Stedman wrote:

> > > There is no general class of problem unless you can provide a single
> > > instance.
> 
> Seriously, I would like to live in your perfect world, but in the past 
> couple of decades I've seen a variety of situations that would have been 
> easier to deal with (or that, in fact, *were* easier to deal with) 
> because of a limiter. 

I only asked for one :-)

> Let's make up some examples:

I didn't want made up examples :-)

> - A piece of hardware on which a program depends goes bad, causing the 
> program to exit immediately upon startup.  Suppose that this program has 
> a high initial startup cost -- so not only is it respawning pointlessly, 
> but it's driving up system load.
> 
> - The disk fills up, causing your X startup to fail.  But because of the 
> continuous respawning, you can't log in on the console!

OK, I'm convinced.

> Exponential back-off would probably be just fine, and represents a 
> fairly common solution to this class of problem.  SysV init simply 
> pauses for a few minutes if the respawn rate exceeds a certain 
> threshold, and then tries again.
> 
> Either behavior would be helpful.

But perhaps both are unnecessarily complicated. Would it not be sufficient 
for runsv to have a configurable "dead time" after starting the supervised 
process before it was again prepared to respawn the child? We want it to 
start a new child quickly, but we don't want it to do so often. So how 
about "within one second", but "at most every ten seconds".

It would also be useful to have a mechanism to distinguish between a
process dying in reponse to a request from runsv, and a program dying
unexpectedly. Perhaps have "finish" and "unexpected_finish" scripts. I'd
certainly like to have a mechanism to run a finish script if a service is
taken down, but not if it just died unexpecedly. The "unexpected_finish"
script could introduce the programmed delay you want, notify the admin,
preserve any essential logs, etc.

---
Charlie