special s6-svscan/perp(d) catch-all logger service option

special s6-svscan/perp(d) catch-all logger service option

[Jeff]

hello,

i am a new subscriber to this mailing list.

i saw that daemontools-encore svscan provides an option to
specify a special catch-all logging service for svscan and its child
supervise processes:

svscan [ directory ] [ log-service ]

If the 'log-service' option is specified and the named subdirectory
exists, svscan starts the service found there and redirects its output
through it. This service is started before any other (since it is the very
important catch-all logger for (among others) svscan's own output).

it would be very nice for s6(-svscan) and perp(d) to provide such
functionality too.

this would simplify starting them directly from init(tab) (or use as init stage 2
in the case of s6) to a great extent as they would do their output redirections
by themselves and also supervise this special catch-all logging service
directly which would init free from this additional task. now init has only 
to supervise (and restart) one process (s6-svscan/perpd) instead of 2
(the additional catch-add logger) which of course is much easier to
achieve.

when using s6-svscan for init stage 2 this would also simplify the
stage 1 script greatly since it can just directly exec into stage 2
now (by using this option) without doing the output redirection
for s6-svscan by itself.

i also think that this can be achieved without too much effort
and since it is an optional feature would not break compatibility
with earlier versions and hence older scripts should still work
without requiring any changes.

kind regards.

Re: special s6-svscan/perp(d) catch-all logger service option

[Laurent Bercot]

[-- Attachment #1: Type: text/plain, Size: 3711 bytes --]

>hello,
>
>i am a new subscriber to this mailing list.

Hi Jeff, and welcome!


>i saw that daemontools-encore svscan provides an option to
>specify a special catch-all logging service for svscan and its child
>supervise processes:
>(...)
>it would be very nice for s6(-svscan) and perp(d) to provide such
>functionality too.

I have good news and bad news.

The bad news is that the way daemontools-encore's svscan manages
its catch-all logger is almost exactly the way I do it in my stage 1
scripts, except it requires ad-hoc code in svscan. So, the same
mechanism, the same functionality, and move the code *into* svscan
instead of *out of* it? yeah, that's gonna be a no for me, dog,
as far as s6 is concerned. (The less code in s6-svscan and
s6-supervise, the better.)

Additionally, the way daemontools-encore does it, the logdir may
or may not be in the scandir. If it is not in the scandir, it will
not be watched by svscan, and if both the supervise process and the
logger die, nothing will ever read the logpipe again and when the
kernel buffer fills up, your supervision tree will eventually freeze.
Whoops.

The good news is that the functionality already exists, can be
implemented without any additional code in s6-svscan, and there
are tools to help you manage the complexity.
It is done by simply having the catch-all logger as a regular
service in the scandir, which ensures it's supervised by svscan,
and performing a little FIFO trickery at svscan start time in order
to redirect its stdout and stderr to a FIFO (that the catch-all
logger will read from). The differences in implementation is that
the logpipe is a FIFO, not an anonymous pipe, and it's held open by
the logger, not by svscan or supervise. Apart from that, it's the
same idea.

You're saying that my implementation makes running s6-svscan under
sysvinit complex because you need 2 lines in /etc/inittab. That is
not true: you only need 1 inittab line, that runs a "mini-stage 1"
script that performs the FIFO trick (as well as any other early
preparation that you need) before executing s6-svscan. If you don't
want to write such a script, you don't have to - you can, for instance,
take inspiration from this one:
https://code.foxkit.us/adelie/packages/blob/8624df0add74c161f045d19b225fb49afd32fa99/system/s6/s6-svscanboot

You're also saying that this implementation makes stage 1 scripts
difficult to write. That is true, and the contents of the above link
prove it: this s6-svscanboot script, and other "stage 1" scripts,
are not easy to come up with unless you're used to juggling with
Unix plumbing.
Fortunately, you don't have to write stage 1 scripts by hand either:
the s6-linux-init package does just that for you.
https://skarnet.org/software/s6-linux-init/

I would recommend you to grab the 0.4.0.1 release of s6-linux-init
and use the stage 1 scripts created by s6-linux-init-maker, either as
is, or as a source of inspiration for your own stage 1 scripts.
Getting a reliable catch-all logger for s6 is *exactly* what
(this version of) s6-linux-init is about.

I would also recommend you, or anyone interested in stage 1 script
creation, to do this sooner rather than later, because a new version
of s6-linux-init is going to be released in less than a week, and it
will be significantly different, and more complex (because more
featureful, with a focus on turnkey sysvinit compatibility, and
palatability for distributions, rather than quick stage 1 script
creation), and stage 1 will be a C program rather than an
execline script, so use 0.4.0.1 if you want a stage 1 script example.

Hope this helps, and good luck!

--
Laurent

s6-svscan catch-all logger service

[Jeff]

26.04.2019, 01:10, "Laurent Bercot" <ska-supervision@skarnet.org>:
> The bad news is that the way daemontools-encore's svscan manages
> its catch-all logger is almost exactly the way I do it in my stage 1
> scripts, except it requires ad-hoc code in svscan.

but this "ad-hoc code" is in C and does not require any execline tricks.
for those to work you need execline installed too.
that way daemontools-encore's svscan can easily get directly started
and supervised by process #1 without any effort.

i think this feature does not require too much "ad-hoc" C code in
the svscan source. it is easy to do that in C while it becomes hard
to impossible do write a shell script that does it (ok, in execline
this seems to be possible, but you need that package installed then).
  
> Additionally, the way daemontools-encore does it, the logdir may
> or may not be in the scandir.

really ? i always used a service subdir of the scandir and it worked
very well.

> If it is not in the scandir, it will
> not be watched by svscan, and if both the supervise process and the
> logger die, nothing will ever read the logpipe again and when the
> kernel buffer fills up, your supervision tree will eventually freeze.

pretty artificial counter example since noone will use a service dir
outside the scandir. i do not know if this is even possible ...
in any case doing so seems very odd to me and was probably
not intended by the author.

> and performing a little FIFO trickery at svscan start time in order

see ? "a little trickery" is necessary here.

> to redirect its stdout and stderr to a FIFO (that the catch-all
> logger will read from). The differences in implementation is that
> the logpipe is a FIFO, not an anonymous pipe, and it's held open by
> the logger, not by svscan or supervise.

using a fifo here is IMO not the best solution when using a simple pipe(2)
would suffice since fifos need read-write access to fs they reside on.

> You're saying that my implementation makes running s6-svscan under
> sysvinit complex because you need 2 lines in /etc/inittab.

that was just a general remark not specific to SysV, busybox or 
toybox init that applies to every process #1 (and also to init stage 1
when using s6-svscan as stage 2)

> That is not true: you only need 1 inittab line, that runs a "mini-stage 1"
> script that performs the FIFO trick (as well as any other early
> preparation that you need) before executing s6-svscan. 

yes that works but introduces an extra step of indirection. and again this
seems to require execline.
it becomes more difficult doing so directly from -say- inittab or its
actual equivalent for the given init system.

> You're also saying that this implementation makes stage 1 scripts
> difficult to write. That is true

indeed.

> I would also recommend you, or anyone interested in stage 1 script
> creation, to do this sooner rather than later, because a new version
> of s6-linux-init is going to be released in less than a week,

i see.

> and it will be significantly different, and more complex
> (because more featureful,
> with a focus on turnkey sysvinit compatibility

who needs such compatibility anyway ?
those who want/need it should run SysV init directly
and start s6 per init script/inittab entry.

> stage 1 will be a C program rather than an execline script

nice ! that sounds really interesting to me.
you have suprised and teased me definitely with this
announcement.

good luck.

Re: s6-svscan catch-all logger service

[Laurent Bercot]

[-- Attachment #1: Type: text/plain, Size: 1728 bytes --]


>i think this feature does not require too much "ad-hoc" C code in
>the svscan source. it is easy to do that in C while it becomes hard
>to impossible do write a shell script that does it (ok, in execline
>this seems to be possible, but you need that package installed then).

You need to have execline installed to run s6 anyway. And again,
you don't have to do the required effort yourself, it has already
been made for you and you can just reuse the existing solutions.


>pretty artificial counter example since noone will use a service dir
>outside the scandir. i do not know if this is even possible ...
>in any case doing so seems very odd to me and was probably
>not intended by the author.

Yes, from what I can see in the code, it works when the logdir is
in the scandir, and it seems to be the intended use. But it also
*appears to work* when the logdir is not in the scandir, and that
is not good. We're talking pid 1 here. It needs to be *airtight*.


>using a fifo here is IMO not the best solution when using a simple pipe(2)
>would suffice since fifos need read-write access to fs they reside on.

Running a supervision tree requires a rw fs anyway, so that's not a
problem at all.


>>  with a focus on turnkey sysvinit compatibility
>who needs such compatibility anyway ?
>those who want/need it should run SysV init directly
>and start s6 per init script/inittab entry.

Distributions do. They don't want to spend hours of effort looking
for places where sysvinit assumptions are made and fixing them, and
that is understandable. Even systemd came with compatibility
halt/poweroff/reboot/shutdown binaries, to make it easier for
distributions to switch.

--
Laurent

catch-all logger service

[Jeff]

26.04.2019, 11:44, "Laurent Bercot" <ska-supervision@skarnet.org>:
> You need to have execline installed to run s6 anyway.

true. but adding the required code to s6-svscan as
daemontools-encore did would obsolete execline at least  for
this purpose entirely (which is a good thing IMO).
it should not be to much of an effort to add this functionality
to s6-svscan (and perp(d) for that).

> Yes, from what I can see in the code, it works when the logdir is
> in the scandir, and it seems to be the intended use. But it also
> *appears to work* when the logdir is not in the scandir, and that
> is not good.

definitely true. but this is non intended use, so don't do it,
it was not recommended in the docs in any way.
(btw: this could be clarified in the doco)
still a pretty artificial consideration.

> We're talking pid 1 here.

not in the case of daemontools-encore which you were referring to.

> It needs to be *airtight*.

indeed, especially excluding any read-write fs access requirements.

> Running a supervision tree requires a rw fs anyway, so that's not a
> problem at all.

yes true, but IMO process #1 should not require read-write access
to any fs for proper operation. for subprocesses this is not a problem,
though.

Re: catch-all logger service

[Laurent Bercot]

>it should not be to much of an effort to add this functionality
>to s6-svscan (and perp(d) for that).

You need to be able to take "no" for an answer.

--
  Laurent

Re: catch-all logger service

[Jeff]

26.04.2019, 20:51, "Laurent Bercot" <ska-supervision@skarnet.org>:
> You need to be able to take "no" for an answer.

i can do that.
and that will be probably the answer i will get from perp's author
aswell (dunno if he reads this list).

it is just important for me to know if this functionality will be added
or not since i have my own runit-init style process #1 implementation
in the makings and do not want to change it when such features
are added to s6-svscan and perpd.

no i know for sure that my code has to do the pipe(2) call and
has to supervise the logging process aswell
(which is a bit more code and one more child process to supervise).

but one could also use runit's "runsv" here which in turn supervises
both: s6-svscan/perpd and the logger.
(same with perp's "rundeux", that fits even better here since it does
not need any service dirs by itself)

but that is a bit awkward since that can be done easily by process #1
itself and hence just adds an additional level of indirection where
it is not really useful/necessary.

and here is another advantage of the daemontools-encore approach:
when given the special logging service option svscan knows it can
run very verbose since it has an associated logger.
(i think daemontools-encore's svscan already operates this way)

this is also important in the case of perpd since its default operation
style is pretty verbose and thus needs a logger.