Re: runit under sysvinit - coping when runsvdir dies

[Charlie Brady <charlieb-supervision@budge.apana.org.au>]

On Mon, 14 Nov 2005, Charles Duffy wrote:

> I recently had a situation on a fielded server where runsvdir (from runit 
> 1.2.3) apparently died; in any event, all the runsv processes which would 
> typically be directly under runsvdir were instead inherited by sysvinit and 
> running there. However, since runsvdir was respawned by sysvinit, a great 
> deal of CPU time was being spent continuously trying to start new runsv 
> instances under the fresh runsvdir -- attempts which failed because there 
> were still runsv instances alive and holding open the relevant locks. I 
> killed the old runsv instances with "runsvctrl e", and fresh children of the 
> new runsvdir took their place -- but there are still some questions raised:
>
> - How could this have happened? The system's message log doesn't show the OOM 
> killer taking down runsvdir or any segfault on the part of the same.

If any of the run scripts (or programs exec'd by the run scripts) did not 
create a new process group, but sent a kill or term signal to its own 
process group (pppd used to do this in the past, and maybe still does), 
then I think it could have brought down the whole pack of cards. In order 
to prevent this, I think that each runsv should start a new process group 
for the run script to execute in.

Note that for any service with a 'down' file, runsv crashing could mean 
that the service changes from an up state to a down state - which is 
almost certainly not want you want to happen (you're running runit to 
prevent such unrequested transitions).