Re: runit/dependencies

Re: runit/dependencies

[Nico Schottelius]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

Gerrit Pape [Mon, Dec 15, 2003 at 10:17:34AM +0100]:
> On Sat, Dec 13, 2003 at 09:57:42PM +0100, nico-linux@schottelius.org wrote:
> > Can you use dependencies in stage 1 and 3, too?
> 
> You can do all you like in stage 1 and 3, they're just shell scripts.

what I meant are dependencies in that may runit does it from itself.
 
> > I really like the style of minit, as it is generally simple, but
> > I like the three stages of runit, although I am missing the
> > dependencies in the first and last stage.
> 
> I've never had the need for this. 

I am for example only starting services if the mounts went well.

> Stage 1 simply prepares the system to
> run services (fsck, mount file systems, ...), stage 3 takes care of
> bringing the services down reliably

but stage3 is _my_ work...and can be faulty.

> and prepares the system for shutdown.
> There's no parallelism in these stage,

why not? I can mount three filesystems parallel.

> and so no need for dependencies
> AFAICS.

We'll see.

> Please discuss runit on the <supervision@list.skarnet.org> mailing list,
> not with me privately.  I prefer public discussions about free software.

Me too.

I even would like to discuss this stuff on the minit mailing list, too,
but I am not a friend of crossposting and perhaps some of the users
of minit are here, too.

What I am exactly looking for is:

- something fast
- something reliable
- something simple

I am currently building up a new Linux distribution and don't want
to use the old style sysVinit, although there are some scripts I wrote
which make bootin quiet fast, too.

Another aspect I am not clear about:

Should *init do supervising? Are the daemontools unecessary in this
case?

Just the need for some information,

Nico

-- 
Keep it simple & stupid, use what's available.
pgp: 8D0E E27A          | Nico Schottelius
http://nerd-hosting.net | http://linux.schottelius.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: runit/dependencies

[Gerrit Pape]

On Thu, Jan 08, 2004 at 12:13:41PM +0100, Nico Schottelius wrote:
> Gerrit Pape [Mon, Dec 15, 2003 at 10:17:34AM +0100]:
> > > I really like the style of minit, as it is generally simple, but
> > > I like the three stages of runit, although I am missing the
> > > dependencies in the first and last stage.
> > I've never had the need for this. 
> 
> I am for example only starting services if the mounts went well.

The runit documentation suggests that mounting filesystems is a stage 1
task.  Services are run in stage 2.  If you make 'mount' a service
that's run in stage 2, other services that depend on a filesystem being
mounted should simply fail if it's not available yet.

> > Stage 1 simply prepares the system to
> > run services (fsck, mount file systems, ...), stage 3 takes care of
> > bringing the services down reliably
> 
> but stage3 is _my_ work...and can be faulty.

You definitely should us the svwaitdown program in stage 3 to ensure
that all service become down cleanly without losing any log entries.
Take a look at the included examples.

> Should *init do supervising? Are the daemontools unecessary in this
> case?

With runit the daemontools package is mostly unnecessary, as runit
implements all key features of the daemontools programs.  The init
replacement (process no 1) doesn't do supervising, the runsv program
started by runsvdir, which is started as stage 2, does.  This modularity
provides you good flexibility of usage.

Regarding all these ``I need support for dependencies'' (I've been asked
often), and even work to provide straight dependencies with runit[0],
I've yet to see a real example that cannot be handled by runit as it is
currently.  It works for me.  All a service needs to do if another
service it depends on isn't available, is to fail.

Regards, Gerrit.

[0] http://article.gmane.org/gmane.comp.sysutils.supervision.general/251

Re: runit/dependencies

[Charlie Brady]

On Mon, 12 Jan 2004, Gerrit Pape wrote:

> Regarding all these ``I need support for dependencies'' (I've been asked
> often), and even work to provide straight dependencies with runit[0],
> I've yet to see a real example that cannot be handled by runit as it is
> currently.  It works for me.  All a service needs to do if another
> service it depends on isn't available, is to fail.

The key word here is "available". The exact semantics of "available" will
vary from service to service. The best that runit can provide is "has been
asked to start and hasn't failed yet". A service run script, OTOH, can
perform arbitrary probes of the "availability" of other services on which
it depends.

I consider the limited semantics of the status provided by "runsvstat" to
be a fatal obstacle to attempts to build dependency management into runit.  
All you can achieve is a slightly ordered parallel startup of multiple
services, with race conditions determining whether or not services start
successfully.

--
Charlie

Re: runit/dependencies

[Gerrit Pape]

On Mon, Jan 12, 2004 at 10:41:15AM -0500, Charlie Brady wrote:
> On Mon, 12 Jan 2004, Gerrit Pape wrote:
> > Regarding all these ``I need support for dependencies'' (I've been asked
> > often), and even work to provide straight dependencies with runit[0],
> > I've yet to see a real example that cannot be handled by runit as it is
> > currently.  It works for me.  All a service needs to do if another
> > service it depends on isn't available, is to fail.
> 
> The key word here is "available". The exact semantics of "available" will
> vary from service to service. The best that runit can provide is "has been
> asked to start and hasn't failed yet". A service run script, OTOH, can
> perform arbitrary probes of the "availability" of other services on which
> it depends.

I completely agree with you.  If a service depends on the availability of
another service, the run script of the dependent service should check
the availability of the service it depends on, and not only if the
service daemon is running (unless the service daemon does it itself).
The run script should abort if the service is unavailable, runsv will
then try to start it again.

Regards, Gerrit.

Re: runit/dependencies

[Kevin P. Fleming]

Gerrit Pape wrote:

> I completely agree with you.  If a service depends on the availability of
> another service, the run script of the dependent service should check
> the availability of the service it depends on, and not only if the
> service daemon is running (unless the service daemon does it itself).
> The run script should abort if the service is unavailable, runsv will
> then try to start it again.

(replying to an old message, forgive me :-)

I've been exploring the daemontools/runit/simpleinit/minit world, and I 
want dependency handling too. However, I want something that I've not 
come across yet, but seems terribly logical to me: if runsv is managing 
a service, and that service dies, and then dies again when runsv tries 
to start it (say, 3 times), then runsv should stop trying to start it 
_AND_ anything that's dependent on it should be told to shutdown.

I can't see a way to do this with runit, since there's no way to back 
into which services are dependent on the one that's failing. minit does 
have enough information to do this, but does not attempt to implement 
this type of behavior.

Has anyone come across a package that already does this?

Re: runit/dependencies

[Alex Efros]

Hi!

On Thu, May 06, 2004 at 07:53:31PM -0700, Kevin P. Fleming wrote:
> I've been exploring the daemontools/runit/simpleinit/minit world, and I 
> want dependency handling too. However, I want something that I've not 
> come across yet, but seems terribly logical to me: if runsv is managing 
> a service, and that service dies, and then dies again when runsv tries 
> to start it (say, 3 times), then runsv should stop trying to start it 
> _AND_ anything that's dependent on it should be told to shutdown.
> 
> I can't see a way to do this with runit, since there's no way to back 
> into which services are dependent on the one that's failing. minit does 
> have enough information to do this, but does not attempt to implement 
> this type of behavior.

You can do this using 'finish' script. It's not easy, but possible.
1) Such 'finish' script should detect what is was executed 3 times in last,
   for example, 5 seconds
2) It should know it's service name (may be hardcoded or calculated from
   current directory name), and then it can execute
    grep 'svwaitup MYNAME' /var/service/*/run
   to find all services which dependent on this one
3) It can then send 'term' signal to these services, send email to admin
   about this failure and execute long 'sleep' before exiting (because after
   it will exit this service will try to start again and all "waiting"
   services may also start again)

> Has anyone come across a package that already does this?

No. Dependencies is a cool feature, but it make things much more complex...
I prefer to have simple and reliable runit without dependency support than
more complex and less reliable runit with dependency support.

-- 
			WBR, Alex.

Re: runit/dependencies

[Kevin P. Fleming]

Alex Efros wrote:

> You can do this using 'finish' script. It's not easy, but possible.
> 1) Such 'finish' script should detect what is was executed 3 times in last,
>    for example, 5 seconds
> 2) It should know it's service name (may be hardcoded or calculated from
>    current directory name), and then it can execute
>     grep 'svwaitup MYNAME' /var/service/*/run
>    to find all services which dependent on this one
> 3) It can then send 'term' signal to these services, send email to admin
>    about this failure and execute long 'sleep' before exiting (because after
>    it will exit this service will try to start again and all "waiting"
>    services may also start again)

This is an interesting technique... Very creative.

> No. Dependencies is a cool feature, but it make things much more complex...
> I prefer to have simple and reliable runit without dependency support than
> more complex and less reliable runit with dependency support.

I'm also considering depinit, which does most of what I want (and 
appears to be simple enough to extend). I'm not sure where the "less 
reliable" part comes into this; why would a program that implements 
dependencies be inherently less reliable? To my mind, svwaitup is 
already an unreliable solution, because dependent services can't 
actually know whether their dependencies are actually up or are just in 
the process of coming up.

Re: runit/dependencies

[Gerrit Pape]

On Thu, May 06, 2004 at 07:53:31PM -0700, Kevin P. Fleming wrote:
> I've been exploring the daemontools/runit/simpleinit/minit world, and I 
> want dependency handling too. However, I want something that I've not 

I'm still curious about an example.  What's the problem that dependency
handling should solve?  What's wrong with?:
 http://smarden.org/runit/dependencies.html

> come across yet, but seems terribly logical to me: if runsv is managing 
> a service, and that service dies, and then dies again when runsv tries 
> to start it (say, 3 times), then runsv should stop trying to start it 

As Alex Efros said, you can do this through the ./finish script.  E.g.:
If ./finish is started the first time, save the time; if started again
within 5 seconds, sleep 2 seconds; if started again within 5 seconds,
sleep 4 seconds; if started again within 5 seconds, run `runsvctrl down .`.
Do you have an example where such a behavior does any good?

> _AND_ anything that's dependent on it should be told to shutdown.

You could use symlinks to save dependency informations: link the service
directories of the dependent services into ./dependents/ and run
`runsvctrl term ./dependents/*`, or `runsvctrl down ./dependents/*`, in
./finish.

Regards, Gerrit.

Re: runit/dependencies

[Charlie Brady]

On Sat, 8 May 2004, Kevin P. Fleming wrote:

> dependencies be inherently less reliable? To my mind, svwaitup is 
> already an unreliable solution, because dependent services can't 
> actually know whether their dependencies are actually up or are just in 
> the process of coming up.

But that is the fatal flaw with *any* dependency scheme implemented by an
init.  The init process cannot know the semantics of every service's
dependencies. If a service really needs to know that some service is fully
available (not just that it has been asked to start), then it (or its run
script actually) must probe for the real availability of that service. 

--
Charlie

Re: runit/dependencies

[Kevin P. Fleming]

Gerrit Pape wrote:

> I'm still curious about an example.  What's the problem that dependency
> handling should solve?  What's wrong with?:
>  http://smarden.org/runit/dependencies.html

I'm not really clear on what you're asking here... the problem I see 
with runit's current dependency handling is that it's embedded into the 
service's start script, which means runit does not know the difference 
between a service that is actually running and one that is waiting for 
some of its dependencies to come up. Also, because runit does not have 
the dependency information itself, the dependent services cannot be 
started in parallel, although that is a minor issue.

> As Alex Efros said, you can do this through the ./finish script.  E.g.:
> If ./finish is started the first time, save the time; if started again
> within 5 seconds, sleep 2 seconds; if started again within 5 seconds,
> sleep 4 seconds; if started again within 5 seconds, run `runsvctrl down .`.
> Do you have an example where such a behavior does any good?

Yes I do; on my servers running Exim (MTA), amavisd-new (virus 
scanning/spam scanning daemon) and clamd (virus scanner daemon), bugs in 
amavisd-new and/or clamd will occasionally cause them to fail and not 
restart. In this situation I would much rather have Exim be shut down 
rather than accept mail it cannot process, because the results of the 
scanning process are used during the SMTP reception phase and if those 
results are not available my only choice is to accept everything and 
scan it later. It is probably possible to configure Exim to do some kind 
of check itself for _every incoming message_ to see if these daemons are 
up, but that makes no sense to me. It also embeds more knowledge into 
the Exim config file than is warranted; Exim does not need to know what 
virus scanners amavisd-new is talking to, but it does need to be stopped 
(IMHO) if those scanners are not available.

> You could use symlinks to save dependency informations: link the service
> directories of the dependent services into ./dependents/ and run
> `runsvctrl term ./dependents/*`, or `runsvctrl down ./dependents/*`, in
> ./finish.

Yes, I have seen this solution before and it appeals to me somewhat, 
although it means duplicating the dependency information in two (or 
more) places.

Re: runit/dependencies

[Kevin P. Fleming]

Charlie Brady wrote:

> But that is the fatal flaw with *any* dependency scheme implemented by an
> init.  The init process cannot know the semantics of every service's
> dependencies. If a service really needs to know that some service is fully
> available (not just that it has been asked to start), then it (or its run
> script actually) must probe for the real availability of that service. 

This is the argument I see repeated multiple times in this list's 
archives: since dependency handling cannot ever be perfect, it shouldn't 
  be used at all.

I disagree, because I think dependency handling can do good things for 
my systems. Given the dependency model implemented by depinit, service A 
that is dependent on service B will not even be started until depinit 
knows that service B's start script successfully finished. runit's 
svwaitup is supposed to accomplish the same thing, but this breaks down 
in a three-service dependency chain because anything dependent on 
service A will think A is already running even though it is sitting in 
svwaitup waiting for B to be up and stable.

Re: runit/dependencies

[George Georgalis]

On Sun, May 09, 2004 at 07:30:10AM -0700, Kevin P. Fleming wrote:
>Yes I do; on my servers running Exim (MTA), amavisd-new (virus 
>scanning/spam scanning daemon) and clamd (virus scanner daemon), bugs in 
>amavisd-new and/or clamd will occasionally cause them to fail and not 
>restart. In this situation I would much rather have Exim be shut down 
>rather than accept mail it cannot process, because the results of the 
>scanning process are used during the SMTP reception phase and if those 
>results are not available my only choice is to accept everything and 

if you use SA in SMTP then wouldn't "spamc -x -c" solve your problem?
With -x your mta should report a temp error and the mail will queue on
remote servers when your spamd is unavailable.

I don't use your other packages but I imaging they can be setup similarly?

Here is an alternate QMAILQUEUE program... it requires a maildir with
qmaild write permissions to "save rejected mails" for inspection. In low
memory systems 20 seconds of sleep can make a big difference. Too few
comments, if parts are unclear just ask.

 # cat /var/qmail/bin/qmail-spamc-queue
#!/bin/bash
echo $0
scq="spamc-queue"
tmp="${scq}/`safecat "${scq}/tmp" "${scq}" </dev/stdin`"
host=`cat control/me`
spamc -x -c <"$tmp"
ec=$?
case $ec in
0) # ham
        sleep 0
        formail -A "X-spamc: ${host}; `date -R`" <"$tmp" \
                | bin/qmail-queue 
        ec=$?
        rm "$tmp"
        exit $ec # whatever qmail-queue exits as
;;
1) # spam 
        sleep 0
        cat "$tmp" | maildir "${scq}" >/dev/null
        rm "$tmp"
        exit 31 # it's spam, mail server permanently refuses
;;
*) # spamc error, 
        echo "Error $0 exit $ec"
        exit 71 # Mail server temporarily refuses to send the message
esac
exit 81 # Internal bug; e.g., segmentation fault.


Regards,
// George

-- 
George Georgalis, Architect and administrator, Linux services. IXOYE
http://galis.org/george/  cell:646-331-2027  mailto:george@galis.org
Key fingerprint = 5415 2738 61CF 6AE1 E9A7  9EF0 0186 503B 9831 1631

Re: runit/dependencies

[Charlie Brady]

On Sun, 9 May 2004, Kevin P. Fleming wrote:

> Charlie Brady wrote:
> 
> > But that is the fatal flaw with *any* dependency scheme implemented by an
> > init.  The init process cannot know the semantics of every service's
> > dependencies. If a service really needs to know that some service is fully
> > available (not just that it has been asked to start), then it (or its run
> > script actually) must probe for the real availability of that service. 
> 
> This is the argument I see repeated multiple times in this list's 
> archives: since dependency handling cannot ever be perfect, it shouldn't 
>   be used at all.

I haven't suggested that dependency handling shouldn't be used. I suggest
that it should be used, but should be used in the run script for a
service, where the semantics of the dependencies can be intimately known.

---
Charlie

Re: runit/dependencies

[Gerrit Pape]

On Sun, May 09, 2004 at 07:30:10AM -0700, Kevin P. Fleming wrote:
> Gerrit Pape wrote:
> >I'm still curious about an example.  What's the problem that dependency
> >handling should solve?  What's wrong with?:
> > http://smarden.org/runit/dependencies.html
> 
> I'm not really clear on what you're asking here... the problem I see 

I'm asking for examples of services that need dependencies implemented
by the init scheme other than how runit already does.  Especially now
that you address multi-level dependencies, like C needs B and B needs A.
What services are A, B, and C, exactly?  Why don't they work with
runit's current service handling?  When you simply start them through
runit, what goes wrong that you want to fix with a dependency concept?

> with runit's current dependency handling is that it's embedded into the 
> service's start script, which means runit does not know the difference 
> between a service that is actually running and one that is waiting for 
> some of its dependencies to come up.

runit cannot know whether the service actually is available even though
it knows that it's running, as already said here.  So there's no benefit
in knowing whether it's waiting or running, runit still cannot know
whether the service is available.

svwaitdown shouldn't be used, as documented, it -at best- is a
workaround, and I think it's misguiding you.  Replace svwaitdown with
your custom check-for-this-service-to-be-up script or program.  E.g.: a
service really needs a syslog service to be available before it's
started; put something like this into the ./run script and all should be
fine (untested):
 lsof /dev/log || exit $?

> >As Alex Efros said, you can do this through the ./finish script.  E.g.:
> >If ./finish is started the first time, save the time; if started again
> >within 5 seconds, sleep 2 seconds; if started again within 5 seconds,
> >sleep 4 seconds; if started again within 5 seconds, run `runsvctrl down .`.
> >Do you have an example where such a behavior does any good?
> 
> Yes I do; on my servers running Exim (MTA), amavisd-new (virus 
> scanning/spam scanning daemon) and clamd (virus scanner daemon), bugs in 
> amavisd-new and/or clamd will occasionally cause them to fail and not 
> restart. In this situation I would much rather have Exim be shut down 
> rather than accept mail it cannot process, because the results of the 
> scanning process are used during the SMTP reception phase and if those 
> results are not available my only choice is to accept everything and 
> scan it later. It is probably possible to configure Exim to do some kind 
> of check itself for _every incoming message_ to see if these daemons are 
> up, but that makes no sense to me. It also embeds more knowledge into 
> the Exim config file than is warranted; Exim does not need to know what 
> virus scanners amavisd-new is talking to, but it does need to be stopped 
> (IMHO) if those scanners are not available.

Obviously exim, when using some other service to scan/process mails, and
this service is unavailable or fails to scan/process the mail message,
should defer acceptance or delivery of a mail message temporarily.  It
doesn't need to ``check itself for _every incoming message_ to see if
these daemons are up'', but it must check if the daemon it contacted
actually did its work.  IMHO the setup should be fixed, and not some
init dependency scheme used as workaround.

Regards, Gerrit.

Re: runit/dependencies

[Kevin P. Fleming]

Gerrit Pape wrote:

> I'm asking for examples of services that need dependencies implemented
> by the init scheme other than how runit already does.  Especially now
> that you address multi-level dependencies, like C needs B and B needs A.
> What services are A, B, and C, exactly?  Why don't they work with
> runit's current service handling?  When you simply start them through
> runit, what goes wrong that you want to fix with a dependency concept?

Sorry for being unclear, I think I've been over-engineering this... In 
any case, the four services in question are exim, amavisd-new, trophie 
and clamd. Exim shouldn't be started until amavisd-new is actually 
running, and amavisd-new shouldn't be started until trophie and clamd 
are both running (and have been up for 10-15 seconds). As best I can 
tell right now, if I put "svwaitup 15 amavisd-new" in exim's start 
script, it will start exim after amavisd-new's start script has been 
running for 15 seconds, even though amavisd-new's start script is still 
waiting for trophie and clamd to be running, so the actual amavisd-new 
daemon is not running yet.

Using the depinit model, where init itself starts the dependencies 
before the depender's start script is run, I don't have this problem.

> runit cannot know whether the service actually is available even though
> it knows that it's running, as already said here.  So there's no benefit
> in knowing whether it's waiting or running, runit still cannot know
> whether the service is available.

I understand your point, but using the depinit mode at least the 
"waiting" state is taken out of the equation. If the service's start 
script has been run and has not exited, then the logical assumption is 
that the service is running. Yes, I know this is imperfect, and it's 
certainly possible that the service process could be running but not 
actually providing the intended services. Without creating "monitor" 
processes to monitor services to see if they are in this condition, I 
don't see how this is a solvable problem.

> svwaitdown shouldn't be used, as documented, it -at best- is a
> workaround, and I think it's misguiding you.  Replace svwaitdown with
> your custom check-for-this-service-to-be-up script or program.  E.g.: a
> service really needs a syslog service to be available before it's
> started; put something like this into the ./run script and all should be
> fine (untested):
>  lsof /dev/log || exit $?

I'm confused; you're referring to svwaitdown not being used this way, 
but then you're talking about a service start script. Did you mean 
svwaitup here instead?

> Obviously exim, when using some other service to scan/process mails, and
> this service is unavailable or fails to scan/process the mail message,
> should defer acceptance or delivery of a mail message temporarily.  It
> doesn't need to ``check itself for _every incoming message_ to see if
> these daemons are up'', but it must check if the daemon it contacted
> actually did its work.  IMHO the setup should be fixed, and not some
> init dependency scheme used as workaround.

I am starting to agree, but I'm a little leery of this situation. I'd 
rather have the "fail safe" of shutting down Exim if I already know that 
the virus scanners are broken, rather than accepting mail and making the 
assumption that amavisd-new has no bugs and will never let mail through 
without scanning it first. The root of the problem here is that Exim has 
absolutely no way to know what amavisd-new did with the message if it 
comes back with a positive response; the message may or may not have 
been scanned. I wish it were not that way but there is not a single 
piece of software on my systems that is completely bug-free :-)

I think I can handle the auto-shutdown down stuff using the finish 
script magic already posted in this thread; if I can find a way to keep 
service A from being started until after B's dependencies (C and D) are 
up and running, I'll be happier. For now I may have no choice but to 
list C and D as dependencies of A as well.

Re: runit/dependencies

[Charlie Brady]

On Mon, 10 May 2004, Kevin P. Fleming wrote:

> are both running (and have been up for 10-15 seconds). As best I can 
> tell right now, if I put "svwaitup 15 amavisd-new" in exim's start 
> script, it will start exim after amavisd-new's start script has been 
> running for 15 seconds, even though amavisd-new's start script is still 
> waiting for trophie and clamd to be running, so the actual amavisd-new 
> daemon is not running yet.

OTOH, if you just exit if amavisd-new's service is not available, then 
supervise will handle this for you.

What you really want to be assured of is that all the services that you 
depend on have been asked to start (which your run script can do be asking 
for all those services to be started) and that the services that you 
depend upon are actually available (which your run script can do by 
testing for them).

> I understand your point, but using the depinit mode at least the 
> "waiting" state is taken out of the equation. If the service's start 
> script has been run and has not exited, then the logical assumption is 
> that the service is running.

But we know that there is always a time period when that assumption is 
false.

> Yes, I know this is imperfect, and it's certainly possible that the
> service process could be running but not actually providing the intended
> services.

There are actually two different states. One is that the service has been 
started, but it not yet providing its service. That's a normal condition, 
especially at system startup. The other is that the service is running, 
but broken. That's a different issue, and not one you should be concerned 
about in dealing with service dependencies and system startup.

> Without creating "monitor"  processes to monitor services to
> see if they are in this condition, I don't see how this is a solvable
> problem.

Sure.

I still don't see the need for anything more than a few "svc -u ..." calls 
in the run script, followed by one or more "probe and exit if service not 
available" calls.

--
Charlie

A: Because we read from top to bottom, left to right.
Q: Why should i start my reply below the quoted text?

Re: runit/dependencies

[Gerrit Pape]

On Mon, May 10, 2004 at 03:57:21PM -0700, Kevin P. Fleming wrote:
> Gerrit Pape wrote:
> >What services are A, B, and C, exactly?  Why don't they work with
> >runit's current service handling?  When you simply start them through
> >runit, what goes wrong that you want to fix with a dependency concept?
> 
> Sorry for being unclear, I think I've been over-engineering this... In 
> any case, the four services in question are exim, amavisd-new, trophie 
> and clamd. Exim shouldn't be started until amavisd-new is actually 
> running, and amavisd-new shouldn't be started until trophie and clamd 
> are both running (and have been up for 10-15 seconds). As best I can 
> tell right now, if I put "svwaitup 15 amavisd-new" in exim's start 
> script, it will start exim after amavisd-new's start script has been 
> running for 15 seconds, even though amavisd-new's start script is still 
> waiting for trophie and clamd to be running, so the actual amavisd-new 
> daemon is not running yet.

You say ``should''; what happens if all the service start at the same
time, don't they work eventually?

> >svwaitdown shouldn't be used, as documented, it -at best- is a
> >workaround, and I think it's misguiding you.  Replace svwaitdown with

> I'm confused; you're referring to svwaitdown not being used this way, 
> but then you're talking about a service start script. Did you mean 
> svwaitup here instead?

Yes, I meant svwaitup, sorry for the confusion.

> I am starting to agree, but I'm a little leery of this situation. I'd 
> rather have the "fail safe" of shutting down Exim if I already know that 
> the virus scanners are broken, rather than accepting mail and making the 
> assumption that amavisd-new has no bugs and will never let mail through 
> without scanning it first. The root of the problem here is that Exim has 
> absolutely no way to know what amavisd-new did with the message if it 
> comes back with a positive response; the message may or may not have 
> been scanned. I wish it were not that way but there is not a single 
> piece of software on my systems that is completely bug-free :-)

Well, this I consider improper inter process communication.  runit
already helps to make ipc more easy and reliable, but this is not a
problem an init scheme should solve.  These services should learn to
communicate with each other if they want to work together.

Regards, Gerrit.

Re: runit/dependencies

[Kevin P. Fleming]

Charlie Brady wrote:

> OTOH, if you just exit if amavisd-new's service is not available, then 
> supervise will handle this for you.

You are correct, I was stuck on the idea that exim's start script should 
only be started once and then do everything needed, but this is not 
necessary. There's no reason runit could not repeatedly re-run exim's 
start script until it finally starts (for the first time during that 
boot of the system).

> But we know that there is always a time period when that assumption is 
> false.

True; it is possible for the start script to have finished executing but 
the service to not actually be available. However, there has to be a 
point at which you stop checking: just because Exim is up and running 
(and will, for example, answer on TCP port 25) does not mean that it is 
ready to do all the things it it supposed to do. In fact, that condition 
can occur after it has already started and there's no way to know that. 
My point is that I'm not sure it makes much sense to bother checking 
whether Exim will answer on port 25 at all, because it doesn't really 
tell you much more than the start script completing without errors told you.

> There are actually two different states. One is that the service has been 
> started, but it not yet providing its service. That's a normal condition, 
> especially at system startup. The other is that the service is running, 
> but broken. That's a different issue, and not one you should be concerned 
> about in dealing with service dependencies and system startup.

Absolutely agree here.

> I still don't see the need for anything more than a few "svc -u ..." calls 
> in the run script, followed by one or more "probe and exit if service not 
> available" calls.

I'm trying to avoid duplication of effort. What if the method that Exim 
uses to communicate with amavisd-new changes in the future? I can update 
exim's config file, but why should I have to change it's service start 
script as well? If I don't it will not have any way to know that 
amavisd-new is running, so it will not start.

I know this probably sounds ridiculously simple and that I shouldn't 
even be worried about the few minutes of effort, but if I'm going to 
change the init process used by all the systems I maintain I'd rather do 
something that's as close to ideal as I can get. Having to make the 
startup scripts aware of how the services communicate with each other 
strikes me as the wrong way to solve the problem.

Re: runit/dependencies

[Kevin P. Fleming]

Charlie Brady wrote:

> I still don't see the need for anything more than a few "svc -u ..." calls 
> in the run script, followed by one or more "probe and exit if service not 
> available" calls.

It appears that your comments are in reference to daemontools, when I 
assumed they were in reference to runit. Does that change anything? I 
think not, given that they operate on a very similar model.

Re: runit/dependencies

[Kevin P. Fleming]

Gerrit Pape wrote:

> You say ``should''; what happens if all the service start at the same
> time, don't they work eventually?

If they all start successfully, yes. That still leaves a window if the 
grandparent service starts quickly and the grandchild services do not, 
during which time (in this case) email can be received and will then sit 
in a queue for later retries, rather then being directly delivered. This 
is not a major problem, of course, but since I see it as being so easily 
avoidable I'd rather avoid it.


> Yes, I meant svwaitup, sorry for the confusion.

So you're saying that I should not use svwaitup at all for checking 
dependent services, but rather should only use some direct means of 
determining whether the service is actually running? If you look at the 
last part of my other reply (to Charlie Brady), you can see why I've 
been resisting this... it puts more details of the service communication 
methods into the start scripts, and I can't see any particular reason 
for it to be there. Just knowing the amavisd-new is listening on its TCP 
port, for example, does not really tell me anything more than knowing 
that its start script did not exit with failure could tell me.

> Well, this I consider improper inter process communication.  runit
> already helps to make ipc more easy and reliable, but this is not a
> problem an init scheme should solve.  These services should learn to
> communicate with each other if they want to work together.

This is not a communication problem, it's a problem caused by the fact 
that software is not bug-free. There is no way for amavisd-new to tell 
exim that it failed to virus scan the message but is passing it along 
anyway; if it knew enough to tell exim that, it would know enough to 
take other evasive action instead. These issues only occur because the 
software is not perfect and will sometimes allow things through without 
scanning them, but only if the scanners have failed after they were 
originally running, and then only under weird circumstances that haven't 
been isolated yet. Given that, and the fact that this type of situation 
could recur any time in the future due to software upgrades, 
configuration errors, etc., I'd much rather have the system just stop 
processing mail rather than risk mail being delivered that says it was 
virus scanned but was not. If I can achieve part (or most) of that goal 
by just shutting down the relevant services when I know the virus 
scanner daemons have either failed (after startup) or never started 
properly to begin with, that gives me a little comfort level I wouldn't 
otherwise have. Does it solve every possible problem? No, of course it 
doesn't :-)

Re: runit/dependencies

[Charlie Brady]

On Tue, 11 May 2004, Kevin P. Fleming wrote:

> Charlie Brady wrote:
> 
> > I still don't see the need for anything more than a few "svc -u ..." calls 
> > in the run script, followed by one or more "probe and exit if service not 
> > available" calls.
> 
> It appears that your comments are in reference to daemontools, when I 
> assumed they were in reference to runit. Does that change anything?

No.

--
Charlie

Re: runit/dependencies

[Alex Efros]

Hi!

On Tue, May 11, 2004 at 11:36:20AM -0700, Kevin P. Fleming wrote:
> This is not a communication problem, it's a problem caused by the fact 
> that software is not bug-free. There is no way for amavisd-new to tell 
> exim that it failed to virus scan the message but is passing it along anyway;

Looks like you try to solve this problem in wrong place.
If your goal is:

> I'd much rather have the system just stop processing mail rather than risk
> mail being delivered that says it was virus scanned but was not.

then is can't be solved by service dependencies. Period.

This problem should be solved by installing some another SMTP and antivirus
software which aware about this issue (or patch your current software).

> If I can achieve part (or most) of that goal 
> by just shutting down the relevant services when I know the virus 
> scanner daemons have either failed (after startup) or never started 
> properly to begin with, that gives me a little comfort level I wouldn't 
> otherwise have. Does it solve every possible problem? No, of course it 
> doesn't :-)

If you want some unreliable workaround this problem using service dependencies,
then it should be enough to just shutdown exim from ./finish scripts for
amavisd and clamd services plus svwaitup for both amavisd and clamd from
./run script for exim.
This is simple, easy, ugly and unreliable solution which should work in most
cases without need to patch/replace your not bug-free software. :-)

-- 
			WBR, Alex.

Re: runit/dependencies

[Charlie Brady]

On Tue, 11 May 2004, Kevin P. Fleming wrote:

> > But we know that there is always a time period when that assumption is 
> > false.
> 
> True; it is possible for the start script to have finished executing but 
> the service to not actually be available.

Not only is it possible, it's always true (for some period).

> However, there has to be a point at which you stop checking: just
> because Exim is up and running (and will, for example, answer on TCP
> port 25) does not mean that it is ready to do all the things it it
> supposed to do.

This just means that you are doing the wrong test for the "availability" 
of the service. Neither your init system nor exim can know exactly which 
of the services provided by exim are required by your service. 

> My point is that I'm not sure it makes much sense to bother checking 
> whether Exim will answer on port 25 at all, because it doesn't really 
> tell you much more than the start script completing without errors told you.

That depends on whether the "answering on port 25" is the feature of exim 
that your program needs. [And it does tell you more than just the start 
script completing. It tells you that exim has proceeded to the point of 
listening on that socket, which might be exactly what you need. Or not.]

> > I still don't see the need for anything more than a few "svc -u ..." calls 
> > in the run script, followed by one or more "probe and exit if service not 
> > available" calls.
> 
> I'm trying to avoid duplication of effort. What if the method that Exim 
> uses to communicate with amavisd-new changes in the future? I can update 
> exim's config file, but why should I have to change it's service start 
> script as well?

Have amavisd-new "export" a probe script for each service it exports. Use 
that script in exim's start script. If amavisd-new changes its exported 
interface, it can change its exported probe script. [But I would guess 
exim will also need to change, to use the new exported interface. You 
can't get away from the fact that when you change the interface between 
two programs, both programs will need to change.]

> I know this probably sounds ridiculously simple and that I shouldn't 
> even be worried about the few minutes of effort, but if I'm going to 
> change the init process used by all the systems I maintain I'd rather do 
> something that's as close to ideal as I can get. Having to make the 
> startup scripts aware of how the services communicate with each other 
> strikes me as the wrong way to solve the problem.

Making init aware of how each pair of programs in the system communicate 
strikes me as altogether the wrong way to solve any problem :-)

---
Charlie

Re: runit/dependencies

[Kevin P. Fleming]

Charlie Brady wrote:

> Have amavisd-new "export" a probe script for each service it exports. Use 
> that script in exim's start script. If amavisd-new changes its exported 
> interface, it can change its exported probe script. [But I would guess 
> exim will also need to change, to use the new exported interface. You 
> can't get away from the fact that when you change the interface between 
> two programs, both programs will need to change.]

Right, both programs need to change, but their service scripts should 
not have to be touched.

Given that I'm coming from the completely unreliability of a 
non-supervised init system now, I think that I can handle going to an 
imperfect dependency-based supervised system. At least I'll know when my 
services are having trouble staying up, and that they'll be restarted if 
at all possible.

Thanks for all the help from everyone who has responded; it sure has 
changed my mind about some things :-)