* Adding a 'setup' phase to service startup? @ 2005-02-21 17:41 Lars Kellogg-Stedman 2005-02-22 4:08 ` Dean Hall ` (2 more replies) 0 siblings, 3 replies; 8+ messages in thread From: Lars Kellogg-Stedman @ 2005-02-21 17:41 UTC (permalink / raw) I was mulling over service dependencies yesterday, and I wonder if it would make sense to add a "setup" phase to runsv's service startup sequence. This would allow, for example, svwaitup to block while a service performed one time initialization. Here I'm thinking particularly of long-running setup actions, such as database initialization or something. The current solution -- waiting for a number of seconds and then crossing our fingers -- doesn't provide the level of control that runit provides in other areas. This would make the start sequence look like this: IF file "setup" exists THEN SET STATE TO S_SETUP RUN file "setup" END IF SET STATE TO S_RUN RUN file "run" IF file "finish" exists THEN SET STATE TO S_FINISH RUN file "finish" END IF svwaitup() would block if a service was in the "setup" phase. This should require only minor changes to runit -- introduce "runsv" to the "setup" script, and modify svwaitup, svwaitdown, and runsvstat to understand a new state value in the status file. -- Lars -- Lars Kellogg-Stedman <lars@oddbit.com> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-21 17:41 Adding a 'setup' phase to service startup? Lars Kellogg-Stedman @ 2005-02-22 4:08 ` Dean Hall 2005-02-22 15:06 ` Thomas Schwinge 2005-02-23 20:07 ` Lars Kellogg-Stedman 2005-02-23 21:12 ` Stefan Karrmann 2005-03-07 11:56 ` Gerrit Pape 2 siblings, 2 replies; 8+ messages in thread From: Dean Hall @ 2005-02-22 4:08 UTC (permalink / raw) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lars Kellogg-Stedman wrote: > I was mulling over service dependencies yesterday, and I wonder if it would > make sense to add a "setup" phase to runsv's service startup sequence. > This would allow, for example, svwaitup to block while a service performed > one time initialization. Although I think runit itself should remain pristine in that it only runs the ./run script for a service, I'm working on a system initialization scheme that factors something just like this in. See my initial draft of the scheme at <http://sourceforge.net/projects/sysinit>; please note that it has changed significantly since the first reference release. In particular, what the relevant part of sysinit does is create an infrastructure for resolving both runit-managed and arbitrary dependencies that does not depend on svwaitup (whose ability to resolve dependencies is fundamentally flawed). Basically, in its current incarnation, sysinit takes the following steps in ./run: 1) It optionally checks a configurable directory for a series of dependencies. A dependency, in this case, is a file in that directory. If the file is a directory (or symlink to one), it is expected to be a runit-managed service; if it is running (using a method described below), the dependency is considered resolved. If the file is an executable file (or a symlink to one), it is run; if it returns 0, the dependency is considered resolved. (That is the part that I think could solve your initialization question.) If any dependency fails, the service initialization is abandoned and can optionally stop the service or simply exit (and allow runit to try again) after an optional delay. 2) If all dependencies resolve, sysinit creates a lockfile for the service (separate from runit's) and starts it. (This is how service dependencies are resolved. Thus, a service is not considered started until all its dependencies are resolved and the service proper is actually running. This does not solve more fundamental race conditions--such as when a service starts but immediately fails, and another service checks its dependency in the short time in which the service is running--but I don't think those are solvable in any straightforward fashion, and I'm not sure if they should, as such race conditions indicate very severe problems that usually require human intervention.) If anyone wants to talk to me about this privately, feel free to email me here. Or we can talk about it on the list if it's okay with Gerrit. I'm open to suggestions about how I can improve my system, and I'd be glad to email out the new implementation (in progress) for inspection and suggestions. Dean > Here I'm thinking particularly of long-running setup actions, such as > database initialization or something. The current solution -- waiting for > a number of seconds and then crossing our fingers -- doesn't provide the > level of control that runit provides in other areas. > > This would make the start sequence look like this: > > IF file "setup" exists THEN > SET STATE TO S_SETUP > RUN file "setup" > END IF > > SET STATE TO S_RUN > RUN file "run" > > IF file "finish" exists THEN > SET STATE TO S_FINISH > RUN file "finish" > END IF > > svwaitup() would block if a service was in the "setup" phase. This should > require only minor changes to runit -- introduce "runsv" to the "setup" > script, and modify svwaitup, svwaitdown, and runsvstat to understand a new > state value in the status file. > > -- Lars > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFCGrBIY5YOge8zySQRAl4YAKCAm5FlsWNOhFZL8EerqkytmOU7fACdHZ+9 To6fK4MQbN/QQzWTvGRNmhc= =sBSs -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-22 4:08 ` Dean Hall @ 2005-02-22 15:06 ` Thomas Schwinge 2005-02-23 20:07 ` Lars Kellogg-Stedman 1 sibling, 0 replies; 8+ messages in thread From: Thomas Schwinge @ 2005-02-22 15:06 UTC (permalink / raw) Cc: supervision On Mon, Feb 21, 2005 at 11:08:41PM -0500, Dean Hall wrote: > 2) If all dependencies resolve, sysinit creates a lockfile for the > service (separate from runit's) and starts it. (This is how service > dependencies are resolved. Thus, a service is not considered started > until all its dependencies are resolved and the service proper is > actually running. This does not solve more fundamental race > conditions--such as when a service starts but immediately fails, and > another service checks its dependency in the short time in which the > service is running--but I don't think those are solvable in any > straightforward fashion, and I'm not sure if they should, as such race > conditions indicate very severe problems that usually require human > intervention.) Just an idea: What about using an executable '/service/S/status' that can be used by a service R to check if the service S is working? This executable's interface would have to be standardized, of course. If '/service/S/status' is not executable, a default of 'svwaitup /service/S' could be used. Example /service/sshd/status #v+ #!/bin/sh # exit 100 if the service can't be running. if svok . || runsvstat . then : else exit 100 fi && # exit 111 if the service might be running, but doesn't accept connections. tcpclient -H -l 0 -R 127.0.0.1 22 true # exit 0 if the service is accepting connections. #v- A service Q, depending on sshd (whatever that might be), could run 'cd /service/S && ./status' and use it's exit code to get aware of S's status. Of course, if that check is done from '/service/Q/run', Q will show up as "running" at the time it is only checking if its dependencies are met. So, every service depending on Q's operativeness would have to check '/service/Q/status' in turn... Perhaps runsv could run e.g. '/service/Q/check' before execing into '/service/Q/run'? '/service/Q/check' could also make sure that Q is not restarted too frequently (Thread "Respawn limit for runsv?"). > If anyone wants to talk to me about this privately, feel free to email > me here. Or we can talk about it on the list if it's okay with Gerrit. I'd say it's on topic here. <URL:http://skarnet.org/lists.html> #v+ * supervision: Discussion about system services, daemon supervision, init, runlevel management, and tools such as daemontools' svscan and supervise, or runit. #v- Regards, Thomas ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-22 4:08 ` Dean Hall 2005-02-22 15:06 ` Thomas Schwinge @ 2005-02-23 20:07 ` Lars Kellogg-Stedman 2005-02-24 4:28 ` Dean Hall 2005-02-24 4:40 ` Dean Hall 1 sibling, 2 replies; 8+ messages in thread From: Lars Kellogg-Stedman @ 2005-02-23 20:07 UTC (permalink / raw) > Although I think runit itself should remain pristine in that it only > runs the ./run script for a service... And the finish script. And the log/run script. And the various control/* scripts. My point being, adding a setup phase to the service doesn't represent much of a change, especially since it has zero impact on the system's behavior if there is no setup script in place. Adding an entire new layer on to handle dependencies would seem to detract from the inherent simplicity of the system -- one should be able to use runsv, by itself, to start up a service and have behave it exactly like it would when run "in production". This means that new logic for things like dependency resolution (or respawn limiting) needs to be either (a) implemented in runsv, or (b) implemented as external commands that can be executed by the 'run' script. Implementing service dependencies in the run script is already difficult, particularly because the runit tools are unable to differentiate between "starting up" and "up". This is the small, reasonably well bounded problem that I'm hoping to solve. -- Lars -- Lars Kellogg-Stedman <lars@oddbit.com> ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-23 20:07 ` Lars Kellogg-Stedman @ 2005-02-24 4:28 ` Dean Hall 2005-02-24 4:40 ` Dean Hall 1 sibling, 0 replies; 8+ messages in thread From: Dean Hall @ 2005-02-24 4:28 UTC (permalink / raw) Lars Kellogg-Stedman wrote: >>Although I think runit itself should remain pristine in that it only >>runs the ./run script for a service... > > > And the finish script. And the log/run script. And the various control/* > scripts. My point being, adding a setup phase to the service doesn't > represent much of a change, especially since it has zero impact on the > system's behavior if there is no setup script in place. I agree. One should be able, completely optionally, to implement whatever dependency resolution scheme one wants--and one can do that right now. Adding complexity to the runit system is only asking for bugs and, perhaps more importantly, code that some people may never use. Runit and its various components all do one thing each, and they do them well, and, IMO, that's the best you can ask of any software. > Adding an entire new layer on to handle dependencies would seem to detract > from the inherent simplicity of the system -- one should be able to use > runsv, by itself, to start up a service and have behave it exactly like it > would when run "in production". This means that new logic for things like > dependency resolution (or respawn limiting) needs to be either (a) > implemented in runsv, or (b) implemented as external commands that can be > executed by the 'run' script. As for (a), implementing it in runsv, I think runsv does its job well now. It doesn't need to do anything else. The daemontools/runit spirit of doing things is to leave room for other tools to do their jobs and let the user decide how complex they want their scheme to be--and of course, for various tools to work together, each of them doing only one thing and doing it well. I find this more in the spirit of Linux than SysV init or many other software tools out there, that either do everything conceivable so as not to leave somebody's obscure desires out in the cold and/or to leave no options to the user (vixie-cron anyone?). > Implementing service dependencies in the run script is already difficult, > particularly because the runit tools are unable to differentiate between > "starting up" and "up". This is the small, reasonably well bounded problem > that I'm hoping to solve. And this is the problem I think I've solved. To be more specific than before, I use several different tools written (for now) in BASH: - sv-isrunning <service_dir>: This determines if a service is running by trying to lock the lockfile created by sv-runlock. If it's locked, the service is running. If not, the service is not running (and may still be checking its own dependencies). - sv-require <service_dir|executable_path>: If the argument is a directory, it asserts that the service is running (using sv-isrunning). If it is the path to an executable file, it runs it (leaving file security to the user); if it returns 0, the dependency resolves; otherwise, the dependency doesn't resolve. - sv-checkdeps <dir>: This checks the directory for service dependencies and runs sv-require (optionally as a different user) on each of them, failing itself on the first failure of sv-require. - sv-runlock: This is a djb-chaining script that merely locks a file and execs its arguments. - sv-run: This is a djb-chaining script that checks dependencies (by calling sv-checkdeps), optionally gets an envdir, and execs run-lock with its arguments. I think it's a pretty simple system that follows the spirit of runit, each piece of the toolkit doing one thing and passing off details to lower-level tools (when necessary). Also, it does solve the problem you're talking about. I've put up a little reference tarball at <http://sysinit.sourceforge.net/> in case anyone wants to take a gander. d > > -- Lars > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-23 20:07 ` Lars Kellogg-Stedman 2005-02-24 4:28 ` Dean Hall @ 2005-02-24 4:40 ` Dean Hall 1 sibling, 0 replies; 8+ messages in thread From: Dean Hall @ 2005-02-24 4:40 UTC (permalink / raw) Lars Kellogg-Stedman wrote: >>Although I think runit itself should remain pristine in that it only >>runs the ./run script for a service... I almost forgot. Here's a sample implementation that I'm currently using for dnscache. This is only one method of using sv-tools, but I think it's an elegant one. Also, I think it's one that leaves the most room open for elegant automation and even (for those who like them) GUI tools. (Please keep in mind that I'm running on gentoo, which is very FHS-compliant; some day I'll break out of the FHS shell (again).) # This envdir sets the environment for sv-run. It tells sv-run not to check dependencies and gives it an envdir to fetch before calling ./run-service. -------- `envdir-print /etc/envdir.d/dnscache/meta`: -------- SV_NO_DEPS='1' SV_RUN_ENVDIR='/etc/envdir.d/dnscache/prerun' -------- # This envdir sets the environment for ./run-service. -------- `envdir-print /etc/envdir.d/dnscache/prerun`: -------- ENVDIR='/etc/envdir.d/dnscache/service' SEED='/var/lib/dnscache-seed' USER='dnscache' -------- # This envdir sets the environment for dnscache itself. -------- `envdir-print /etc/envdir.d/dnscache/service`: -------- CACHESIZE='10000000' DATALIMIT='30000000' IP='192.168.0.53' IPSEND='0.0.0.0' ROOT='/etc/service.d/all/dnscache/root' -------- # Obviously, this gets run first: -------- ./run: -------- #!/bin/sh # PATH='/usr/sbin:/usr/bin:/sbin:/bin' exec 2>&1 exec \ envdir /etc/envdir.d/dnscache/meta \ sv-run -------- # After checking dependencies and such, sv-run runs this: -------- ./run-service: -------- #!/bin/sh # # Requires: # - USER # - SEED # PATH='/usr/sbin:/usr/bin:/sbin:/bin' # Create a new random seed. dd if=/dev/urandom of="$SEED" count=1 bs=128 > /dev/null 2>&1 exec 2>&1 exec < "$SEED" exec \ env - PATH="$PATH" \ envuidgid "$USER" \ envdir "$ENVDIR" \ dnscache -------- d ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-21 17:41 Adding a 'setup' phase to service startup? Lars Kellogg-Stedman 2005-02-22 4:08 ` Dean Hall @ 2005-02-23 21:12 ` Stefan Karrmann 2005-03-07 11:56 ` Gerrit Pape 2 siblings, 0 replies; 8+ messages in thread From: Stefan Karrmann @ 2005-02-23 21:12 UTC (permalink / raw) My 2 cent: Lars Kellogg-Stedman (Mon, Feb 21, 2005 at 05:41:59PM +0000): > I was mulling over service dependencies yesterday, and I wonder if it would > make sense to add a "setup" phase to runsv's service startup sequence. > This would allow, for example, svwaitup to block while a service performed > one time initialization. You may use pipe-tools, cf. <http://www.skarnet.org/software/pipe-tools/index.html>. > Here I'm thinking particularly of long-running setup actions, such as > database initialization or something. The current solution -- waiting for > a number of seconds and then crossing our fingers -- doesn't provide the > level of control that runit provides in other areas. > > This would make the start sequence look like this: > > IF file "setup" exists THEN > SET STATE TO S_SETUP > RUN file "setup" > END IF > > SET STATE TO S_RUN > RUN file "run" > > IF file "finish" exists THEN > SET STATE TO S_FINISH > RUN file "finish" > END IF With pipe-tools you only need the run and finish file: ######################### # run file ./setup npt-notify ./events isup standard-run-script ######################### # finish file npt-notify ./events down ######################### The setup code my wait on another service X by ,npt-wait /service/X/events'. The demon can even use libnptrigger to signal the events during its running. Regards, -- Stefan Karrmann ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Adding a 'setup' phase to service startup? 2005-02-21 17:41 Adding a 'setup' phase to service startup? Lars Kellogg-Stedman 2005-02-22 4:08 ` Dean Hall 2005-02-23 21:12 ` Stefan Karrmann @ 2005-03-07 11:56 ` Gerrit Pape 2 siblings, 0 replies; 8+ messages in thread From: Gerrit Pape @ 2005-03-07 11:56 UTC (permalink / raw) On Mon, Feb 21, 2005 at 05:41:59PM +0000, Lars Kellogg-Stedman wrote: > I was mulling over service dependencies yesterday, and I wonder if it would > make sense to add a "setup" phase to runsv's service startup sequence. > This would allow, for example, svwaitup to block while a service performed > one time initialization. > > Here I'm thinking particularly of long-running setup actions, such as > database initialization or something. The current solution -- waiting for > a number of seconds and then crossing our fingers -- doesn't provide the > level of control that runit provides in other areas. Yes, true. I don't consider svwaitup as the current solution, but as a workaround. The fix actually should be in the service daemons I think, see http://smarden.org/runit/dependencies.html > This would make the start sequence look like this: [...] > svwaitup() would block if a service was in the "setup" phase. This should > require only minor changes to runit -- introduce "runsv" to the "setup" > script, and modify svwaitup, svwaitdown, and runsvstat to understand a new > state value in the status file. This doesn't eliminate the race condition, just lets it look a bit better. I'm still looking for examples why service dependencies, more than runit currently provides, is needed. What are these services that must be started not before another service has been started and is available, and that don't recognize the error if the service depended upon isn't available? How do these services behave on that error condition?, shouldn't they log the error and fail? What are these services that need to be taken down manually after a service they depend upon became down? How do they behave if a service they need has been available on startup, but disappears while uptime?, shouldn't they log the error and fail, or maybe retry? What other problems are service dependencies meant to solve? I'm aware of one example, the problem that service daemons that log through syslog cannot know whether the syslog service is enabled, simply because the corresponding unix functions return void. To work around this for services that cannot switch to the runit logging facility, I added the socklog-check program to the socklog package. This program checks for a listener on (e.g.) /dev/log, and fails if the socket isn't maintained; it can be added to the top of the run script with set -e or as `socklog-check unix /dev/log || exit 1` to prevent the service from being started in this case. Thanks, Gerrit. ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2005-03-07 11:56 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2005-02-21 17:41 Adding a 'setup' phase to service startup? Lars Kellogg-Stedman 2005-02-22 4:08 ` Dean Hall 2005-02-22 15:06 ` Thomas Schwinge 2005-02-23 20:07 ` Lars Kellogg-Stedman 2005-02-24 4:28 ` Dean Hall 2005-02-24 4:40 ` Dean Hall 2005-02-23 21:12 ` Stefan Karrmann 2005-03-07 11:56 ` Gerrit Pape
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).