Adding a 'setup' phase to service startup?

Adding a 'setup' phase to service startup?

[Lars Kellogg-Stedman]

I was mulling over service dependencies yesterday, and I wonder if it would
make sense to add a "setup" phase to runsv's service startup sequence.
This would allow, for example, svwaitup to block while a service performed
one time initialization.

Here I'm thinking particularly of long-running setup actions, such as
database initialization or something.  The current solution -- waiting for
a number of seconds and then crossing our fingers -- doesn't provide the
level of control that runit provides in other areas.

This would make the start sequence look like this:

  IF file "setup" exists THEN
    SET STATE TO S_SETUP
    RUN file "setup"
  END IF

  SET STATE TO S_RUN
  RUN file "run"

  IF file "finish" exists THEN
    SET STATE TO S_FINISH
    RUN file "finish"
  END IF

svwaitup() would block if a service was in the "setup" phase.  This should
require only minor changes to runit -- introduce "runsv" to the "setup"
script, and modify svwaitup, svwaitdown, and runsvstat to understand a new
state value in the status file.

-- Lars

-- 
Lars Kellogg-Stedman <lars@oddbit.com>

Re: Adding a 'setup' phase to service startup?

[Dean Hall]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lars Kellogg-Stedman wrote:
> I was mulling over service dependencies yesterday, and I wonder if it would
> make sense to add a "setup" phase to runsv's service startup sequence.
> This would allow, for example, svwaitup to block while a service performed
> one time initialization.

Although I think runit itself should remain pristine in that it only
runs the ./run script for a service, I'm working on a system
initialization scheme that factors something just like this in. See my
initial draft of the scheme at
<http://sourceforge.net/projects/sysinit>; please note that it has
changed significantly since the first reference release.

In particular, what the relevant part of sysinit does is create an
infrastructure for resolving both runit-managed and arbitrary
dependencies that does not depend on svwaitup (whose ability to resolve
dependencies is fundamentally flawed).

Basically, in its current incarnation, sysinit takes the following steps
in ./run:

1) It optionally checks a configurable directory for a series of
dependencies. A dependency, in this case, is a file in that directory.
If the file is a directory (or symlink to one), it is expected to be a
runit-managed service; if it is running (using a method described
below), the dependency is considered resolved. If the file is an
executable file (or a symlink to one), it is run; if it returns 0, the
dependency is considered resolved. (That is the part that I think could
solve your initialization question.) If any dependency fails, the
service initialization is abandoned and can optionally stop the service
or simply exit (and allow runit to try again) after an optional delay.

2) If all dependencies resolve, sysinit creates a lockfile for the
service (separate from runit's) and starts it. (This is how service
dependencies are resolved. Thus, a service is not considered started
until all its dependencies are resolved and the service proper is
actually running. This does not solve more fundamental race
conditions--such as when a service starts but immediately fails, and
another service checks its dependency in the short time in which the
service is running--but I don't think those are solvable in any
straightforward fashion, and I'm not sure if they should, as such race
conditions indicate very severe problems that usually require human
intervention.)

If anyone wants to talk to me about this privately, feel free to email
me here. Or we can talk about it on the list if it's okay with Gerrit.
I'm open to suggestions about how I can improve my system, and I'd be
glad to email out the new implementation (in progress) for inspection
and suggestions.


Dean

> Here I'm thinking particularly of long-running setup actions, such as
> database initialization or something.  The current solution -- waiting for
> a number of seconds and then crossing our fingers -- doesn't provide the
> level of control that runit provides in other areas.
> 
> This would make the start sequence look like this:
> 
>   IF file "setup" exists THEN
>     SET STATE TO S_SETUP
>     RUN file "setup"
>   END IF
> 
>   SET STATE TO S_RUN
>   RUN file "run"
> 
>   IF file "finish" exists THEN
>     SET STATE TO S_FINISH
>     RUN file "finish"
>   END IF
> 
> svwaitup() would block if a service was in the "setup" phase.  This should
> require only minor changes to runit -- introduce "runsv" to the "setup"
> script, and modify svwaitup, svwaitdown, and runsvstat to understand a new
> state value in the status file.
> 
> -- Lars
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCGrBIY5YOge8zySQRAl4YAKCAm5FlsWNOhFZL8EerqkytmOU7fACdHZ+9
To6fK4MQbN/QQzWTvGRNmhc=
=sBSs
-----END PGP SIGNATURE-----

Re: Adding a 'setup' phase to service startup?

[Thomas Schwinge]

On Mon, Feb 21, 2005 at 11:08:41PM -0500, Dean Hall wrote:
> 2) If all dependencies resolve, sysinit creates a lockfile for the
> service (separate from runit's) and starts it. (This is how service
> dependencies are resolved. Thus, a service is not considered started
> until all its dependencies are resolved and the service proper is
> actually running. This does not solve more fundamental race
> conditions--such as when a service starts but immediately fails, and
> another service checks its dependency in the short time in which the
> service is running--but I don't think those are solvable in any
> straightforward fashion, and I'm not sure if they should, as such race
> conditions indicate very severe problems that usually require human
> intervention.)

Just an idea:
What about using an executable '/service/S/status' that can be used by a
service R to check if the service S is working?
This executable's interface would have to be standardized, of course.
If '/service/S/status' is not executable, a default of
'svwaitup /service/S' could be used.

Example /service/sshd/status
#v+
#!/bin/sh

# exit 100 if the service can't be running.
if svok . ||
   runsvstat .
then :
else exit 100
fi &&

# exit 111 if the service might be running, but doesn't accept connections.
tcpclient -H -l 0 -R 127.0.0.1 22 true

# exit 0 if the service is accepting connections.
#v-

A service Q, depending on sshd (whatever that might be), could run
'cd /service/S && ./status' and use it's exit code to get aware of S's
status.
Of course, if that check is done from '/service/Q/run', Q will show up
as "running" at the time it is only checking if its dependencies are
met.
So, every service depending on Q's operativeness would have to check
'/service/Q/status' in turn...

Perhaps runsv could run e.g. '/service/Q/check' before execing into
'/service/Q/run'?
'/service/Q/check' could also make sure that Q is not restarted too
frequently (Thread "Respawn limit for runsv?").


> If anyone wants to talk to me about this privately, feel free to email
> me here. Or we can talk about it on the list if it's okay with Gerrit.

I'd say it's on topic here.
<URL:http://skarnet.org/lists.html>
#v+
* supervision: Discussion about system services, daemon supervision, init, runlevel management, and tools such as daemontools' svscan and supervise, or runit. 
#v-


Regards,
 Thomas

Re: Adding a 'setup' phase to service startup?

[Lars Kellogg-Stedman]

> Although I think runit itself should remain pristine in that it only
> runs the ./run script for a service...

And the finish script.  And the log/run script.  And the various control/*
scripts.  My point being, adding a setup phase to the service doesn't
represent much of a change, especially since it has zero impact on the
system's behavior if there is no setup script in place.

Adding an entire new layer on to handle dependencies would seem to detract
from the inherent simplicity of the system -- one should be able to use
runsv, by itself, to start up a service and have behave it exactly like it
would when run "in production".  This means that new logic for things like
dependency resolution (or respawn limiting) needs to be either (a)
implemented in runsv, or (b) implemented as external commands that can be
executed by the 'run' script.

Implementing service dependencies in the run script is already difficult,
particularly because the runit tools are unable to differentiate between
"starting up" and "up".  This is the small, reasonably well bounded problem
that I'm hoping to solve.

-- Lars

-- 
Lars Kellogg-Stedman <lars@oddbit.com>

Re: Adding a 'setup' phase to service startup?

[Stefan Karrmann]

My 2 cent:

Lars Kellogg-Stedman (Mon, Feb 21, 2005 at 05:41:59PM +0000):
> I was mulling over service dependencies yesterday, and I wonder if it would
> make sense to add a "setup" phase to runsv's service startup sequence.
> This would allow, for example, svwaitup to block while a service performed
> one time initialization.

You may use pipe-tools, cf.
<http://www.skarnet.org/software/pipe-tools/index.html>.

> Here I'm thinking particularly of long-running setup actions, such as
> database initialization or something.  The current solution -- waiting for
> a number of seconds and then crossing our fingers -- doesn't provide the
> level of control that runit provides in other areas.
> 
> This would make the start sequence look like this:
> 
>   IF file "setup" exists THEN
>     SET STATE TO S_SETUP
>     RUN file "setup"
>   END IF
> 
>   SET STATE TO S_RUN
>   RUN file "run"
> 
>   IF file "finish" exists THEN
>     SET STATE TO S_FINISH
>     RUN file "finish"
>   END IF

With pipe-tools you only need the run and finish file:

#########################
# run file
./setup
npt-notify ./events isup
standard-run-script

#########################
# finish file
npt-notify ./events down
#########################

The setup code my wait on another service X by ,npt-wait /service/X/events'.
The demon can even use libnptrigger to signal the events during its running.

Regards,
-- 
Stefan Karrmann

Re: Adding a 'setup' phase to service startup?

[Dean Hall]

Lars Kellogg-Stedman wrote:
>>Although I think runit itself should remain pristine in that it only
>>runs the ./run script for a service...
> 
> 
> And the finish script.  And the log/run script.  And the various control/*
> scripts.  My point being, adding a setup phase to the service doesn't
> represent much of a change, especially since it has zero impact on the
> system's behavior if there is no setup script in place.

I agree. One should be able, completely optionally, to implement
whatever dependency resolution scheme one wants--and one can do that
right now. Adding complexity to the runit system is only asking for bugs
and, perhaps more importantly, code that some people may never use.
Runit and its various components all do one thing each, and they do them
well, and, IMO, that's the best you can ask of any software.

> Adding an entire new layer on to handle dependencies would seem to detract
> from the inherent simplicity of the system -- one should be able to use
> runsv, by itself, to start up a service and have behave it exactly like it
> would when run "in production".  This means that new logic for things like
> dependency resolution (or respawn limiting) needs to be either (a)
> implemented in runsv, or (b) implemented as external commands that can be
> executed by the 'run' script.

As for (a), implementing it in runsv, I think runsv does its job well
now. It doesn't need to do anything else. The daemontools/runit spirit
of doing things is to leave room for other tools to do their jobs and
let the user decide how complex they want their scheme to be--and of
course, for various tools to work together, each of them doing only one
thing and doing it well.

I find this more in the spirit of Linux than SysV init or many other
software tools out there, that either do everything conceivable so as
not to leave somebody's obscure desires out in the cold and/or to leave
no options to the user (vixie-cron anyone?).

> Implementing service dependencies in the run script is already difficult,
> particularly because the runit tools are unable to differentiate between
> "starting up" and "up".  This is the small, reasonably well bounded problem
> that I'm hoping to solve.

And this is the problem I think I've solved. To be more specific than
before, I use several different tools written (for now) in BASH:

- sv-isrunning <service_dir>:
This determines if a service is running by trying to lock the lockfile
created by sv-runlock. If it's locked, the service is running. If not,
the service is not running (and may still be checking its own dependencies).
- sv-require <service_dir|executable_path>:
If the argument is a directory, it asserts that the service is running
(using sv-isrunning). If it is the path to an executable file, it runs
it (leaving file security to the user); if it returns 0, the dependency
resolves; otherwise, the dependency doesn't resolve.
- sv-checkdeps <dir>:
This checks the directory for service dependencies and runs sv-require
(optionally as a different user) on each of them, failing itself on the
first failure of sv-require.
- sv-runlock:
This is a djb-chaining script that merely locks a file and execs its
arguments.
- sv-run:
This is a djb-chaining script that checks dependencies (by calling
sv-checkdeps), optionally gets an envdir, and execs run-lock with its
arguments.

I think it's a pretty simple system that follows the spirit of runit,
each piece of the toolkit doing one thing and passing off details to
lower-level tools (when necessary). Also, it does solve the problem
you're talking about.

I've put up a little reference tarball at
<http://sysinit.sourceforge.net/> in case anyone wants to take a gander.


d

> 
> -- Lars
> 

Re: Adding a 'setup' phase to service startup?

[Dean Hall]

Lars Kellogg-Stedman wrote:
>>Although I think runit itself should remain pristine in that it only
>>runs the ./run script for a service...

I almost forgot. Here's a sample implementation that I'm currently using
for dnscache. This is only one method of using sv-tools, but I think
it's an elegant one. Also, I think it's one that leaves the most room
open for elegant automation and even (for those who like them) GUI tools.

(Please keep in mind that I'm running on gentoo, which is very
FHS-compliant; some day I'll break out of the FHS shell (again).)


# This envdir sets the environment for sv-run. It tells sv-run not to
check dependencies and gives it an envdir to fetch before calling
./run-service.
--------
`envdir-print /etc/envdir.d/dnscache/meta`:
--------
SV_NO_DEPS='1'
SV_RUN_ENVDIR='/etc/envdir.d/dnscache/prerun'
--------

# This envdir sets the environment for ./run-service.
--------
`envdir-print /etc/envdir.d/dnscache/prerun`:
--------
ENVDIR='/etc/envdir.d/dnscache/service'
SEED='/var/lib/dnscache-seed'
USER='dnscache'
--------

# This envdir sets the environment for dnscache itself.
--------
`envdir-print /etc/envdir.d/dnscache/service`:
--------
CACHESIZE='10000000'
DATALIMIT='30000000'
IP='192.168.0.53'
IPSEND='0.0.0.0'
ROOT='/etc/service.d/all/dnscache/root'
--------

# Obviously, this gets run first:
--------
./run:
--------
#!/bin/sh
#
PATH='/usr/sbin:/usr/bin:/sbin:/bin'

exec 2>&1
exec \
        envdir /etc/envdir.d/dnscache/meta \
        sv-run
--------

# After checking dependencies and such, sv-run runs this:
--------
./run-service:
--------
#!/bin/sh
#
# Requires:
# - USER
# - SEED
#
PATH='/usr/sbin:/usr/bin:/sbin:/bin'

# Create a new random seed.
dd if=/dev/urandom of="$SEED" count=1 bs=128 > /dev/null 2>&1

exec 2>&1
exec < "$SEED"
exec \
        env - PATH="$PATH" \
        envuidgid "$USER" \
        envdir "$ENVDIR" \
        dnscache
--------


d

Re: Adding a 'setup' phase to service startup?

[Gerrit Pape]

On Mon, Feb 21, 2005 at 05:41:59PM +0000, Lars Kellogg-Stedman wrote:
> I was mulling over service dependencies yesterday, and I wonder if it would
> make sense to add a "setup" phase to runsv's service startup sequence.
> This would allow, for example, svwaitup to block while a service performed
> one time initialization.
> 
> Here I'm thinking particularly of long-running setup actions, such as
> database initialization or something.  The current solution -- waiting for
> a number of seconds and then crossing our fingers -- doesn't provide the
> level of control that runit provides in other areas.

Yes, true.  I don't consider svwaitup as the current solution, but as a
workaround.  The fix actually should be in the service daemons I think,
see http://smarden.org/runit/dependencies.html

> This would make the start sequence look like this:
[...]
> svwaitup() would block if a service was in the "setup" phase.  This should
> require only minor changes to runit -- introduce "runsv" to the "setup"
> script, and modify svwaitup, svwaitdown, and runsvstat to understand a new
> state value in the status file.

This doesn't eliminate the race condition, just lets it look a bit
better.

I'm still looking for examples why service dependencies, more than runit
currently provides, is needed.  What are these services that must be
started not before another service has been started and is available,
and that don't recognize the error if the service depended upon isn't
available?  How do these services behave on that error condition?,
shouldn't they log the error and fail?

What are these services that need to be taken down manually after a
service they depend upon became down?  How do they behave if a service
they need has been available on startup, but disappears while uptime?,
shouldn't they log the error and fail, or maybe retry?

What other problems are service dependencies meant to solve?

I'm aware of one example, the problem that service daemons that log
through syslog cannot know whether the syslog service is enabled, simply
because the corresponding unix functions return void.  To work around
this for services that cannot switch to the runit logging facility, I
added the socklog-check program to the socklog package.  This program
checks for a listener on (e.g.) /dev/log, and fails if the socket isn't
maintained; it can be added to the top of the run script with set -e or
as `socklog-check unix /dev/log || exit 1` to prevent the service from
being started in this case.

Thanks, Gerrit.