S6 Queries

S6 Queries

[Arjun D R]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

Hi Team,

We are trying to migrate from systemd init system to S6. We have a few
queries, please help us on the same.

1. In systemd, the services are grouped as targets and each target depends
on another target as well. They start as targets. [ex: Reached
local-fs.target, Reached network.target, Reached UI target,...]. Is there
any way in S6 to start the init system based on bundles?

2. Are there any ways to have loosely coupling dependencies? In systemd, we
have After=. After option will help the current service to start after the
mentioned service (in after). And the current service will anyway start
even if the mentioned service in After fails to start. Do we have such
loosely coupled dependency facility in S6?

3. Is there any tool available in S6 to measure the time taken by each
service to start? We can manually measure it from the logs, but still
looking for a tool which can provide accurate data.

4. Does the S6 init system provide better boot up performance compared to
systemd ?  One of our main motives is to attain better bootup performance.
Is our expectation correct?

Thanks in advance,
Arjun

Re: S6 Queries

[Laurent Bercot]

>1. In systemd, the services are grouped as targets and each target depends
>on another target as well. They start as targets. [ex: Reached
>local-fs.target, Reached network.target, Reached UI target,...]. Is there
>any way in S6 to start the init system based on bundles?

  Yes, that is what bundles are for. In your stage 2 boot script
(typically /etc/s6-linux-init/current/scripts/rc.init), you should
invoke s6-rc as:
   s6-rc change top
if "top" is the name of your default bundle, i.e. the bundle that
contains all the services you want to start at boot time. You can
basically convert the contents of your systemd targets directly into
contents of your s6-rc bundles; and you decide which one will be
brought up at boot time via the s6-rc invocation in your stage 2
init script.


>2. Are there any ways to have loosely coupling dependencies? In systemd, we
>have After=. After option will help the current service to start after the
>mentioned service (in after). And the current service will anyway start
>even if the mentioned service in After fails to start. Do we have such
>loosely coupled dependency facility in S6?

  Not at the moment, no. The next version of s6-rc will allow more types
of dependencies, with clearer semantics than the systemd ones (After=, 
Requires= and Wants= are not orthogonal, which is unintuitive and causes
misuse); but it is still in early development.

  For now, s6-rc only provides one type of dependency, which is the
equivalent of Requires+After. I realize this is not flexible enough
for a lot of real use cases, which is one of the reasons why another
version is in development. :)


>3. Is there any tool available in S6 to measure the time taken by each
>service to start? We can manually measure it from the logs, but still
>looking for a tool which can provide accurate data.

  Honestly, if you use the -v2 option to your s6-rc invocation, as in
   s6-rc -v2 change top
and you ask the catch-all logger to timestamp its lines (which should
be the default, but you can change the timestamp style via the -t
option to s6-linux-init-maker)
then the difference of timestamps between the lines:
   s6-rc: info: service foo: starting
and
   s6-rc: info: service foo successfully started
will give you a pretty accurate measurement of the time it took service
foo to start. These lines are written by s6-rc exactly as the
"starting" or "completed" event occurs, and they are timestamped by
s6-log immediately; the code path is the same for both events, so the
delays cancel out, and the only inaccuracy left is randomness due to
scheduling, which should not be statistically significant.

  At the moment, the s6-rc log is the easiest place to get this data
from. You could probably hack something with the "time" shell command
and s6-svwait, such as
   s6-svwait -u /run/service/foo ; time s6-svwait -U /run/service/foo
which would give you the time it took for foo to become ready; but
I doubt it would be any more accurate than using the timestamps in the
s6-rc logs, and it's really not convenient to set up.


>4. Does the S6 init system provide better boot up performance compared to
>systemd ?  One of our main motives is to attain better bootup performance.
>Is our expectation correct?

  The boot up performance should be more or less *similar* to systemd.
The code paths used by the s6 ecosystem are much shorter than the ones
used by systemd, so _in theory_ you should get faster boots with s6.

  However, systemd cheats, by starting services before their dependencies
are ready. For instance, it will start services before any kind of
logging is ready, which is pretty dangerous for several reasons. As a
part of its "socket activation" thing, it will also claim readiness
on some sockets before even attempting to run the actual serving
processes (which may fail, in which case readiness was a lie.)
Because of that, when everything goes well, systemd cuts corners that
s6 does not, and may gain some advantage.

  So all in all, I expect that depending on your system, the difference
in speed will not be remarkable. On very simple setups (just a few
services), systemd's overhead may be noticeable and you may see real
improvements with s6. On complex setups with lots of dependencies, s6
might still have the speed advantage but I don't think it will be
anything amazing. The real benefit of s6 is that it achieves
roughly the same speed as systemd *while being more reliable and
predictable*.

  If you actually make bootup speed comparisons between systemd and s6,
please share them! I am interested in that kind of benchmarks, and
I'm sure the community would like to see numbers as well.

--
  Laurent

Re: S6 Queries

[Steve Litt]

Laurent Bercot said on Mon, 02 Aug 2021 08:27:30 +0000


>>2. Are there any ways to have loosely coupling dependencies? In
>>systemd, we have After=. After option will help the current service
>>to start after the mentioned service (in after). And the current
>>service will anyway start even if the mentioned service in After
>>fails to start. Do we have such loosely coupled dependency facility
>>in S6?  
>
>  Not at the moment, no. 

I thought the way to do what the OP asked is:

=============================
#!/bin/sh
s6-svc -u myrequirement || exit 1
exec mydaemon -myarg1 -myarg2
=============================


SteveT

Steve Litt 
Spring 2021 featured book: Troubleshooting Techniques of the Successful
Technologist http://www.troubleshooters.com/techniques

Re: S6 Queries

[Laurent Bercot]

>I thought the way to do what the OP asked is:
>
>=============================
>#!/bin/sh
>s6-svc -u myrequirement || exit 1
>exec mydaemon -myarg1 -myarg2
>=============================

  This is not a good idea in a s6-rc installation, because it sends
raw s6 commands, which may mess with the service state as viewed by
s6-rc. Also, it sends control commands to a service from another
service's run script, which is bad form in general: it is
unintuitive that starting the mydaemon service also causes the
myrequirement service to be started. Dependencies should be handled at
the service manager level, not at the process supervision level.

  Of course, you can do that in a pinch for small s6-only installations,
where you have no proper dependency engine, but that does not seem to be
what the OP is asking.

  And even then, this does not implement After= because s6-svc -u
returns instantly. This only implements Wants=. To implement After=,
you would need something like s6-svc -uwU instead, which is not good
because it adds the myrequirement readiness delay to the mydaemon
readiness delay, so if mydaemon has strict readiness timeouts, it
can make it fail.

  All in all, it's better to avoid controlling another service in a run
script: there's always an annoying corner case somewhere. Dependency
management is best handled by the external tool that is explicitly
supposed to do it: the service manager.

--
  Laurent

Re: S6 Queries

[Steve Litt]

Laurent Bercot said on Mon, 02 Aug 2021 19:39:47 +0000

>>I thought the way to do what the OP asked is:
>>
>>=============================
>>#!/bin/sh
>>s6-svc -u myrequirement || exit 1
>>exec mydaemon -myarg1 -myarg2
>>=============================  
>
>  This is not a good idea in a s6-rc installation, because it sends
>raw s6 commands, which may mess with the service state as viewed by
>s6-rc. Also, it sends control commands to a service from another
>service's run script, which is bad form in general: it is
>unintuitive that starting the mydaemon service also causes the
>myrequirement service to be started. Dependencies should be handled at
>the service manager level, not at the process supervision level.

Do you think this is any better?

=============================
#!/bin/sh
test_for_myrequirement || exit 1
exec mydaemon -myarg1 -myarg2
=============================

SteveT

Steve Litt 
Spring 2021 featured book: Troubleshooting Techniques of the Successful
Technologist http://www.troubleshooters.com/techniques

Re: S6 Queries

[Laurent Bercot]

>Do you think this is any better?
>
>=============================
>#!/bin/sh
>test_for_myrequirement || exit 1
>exec mydaemon -myarg1 -myarg2
>=============================

  This does not accomplish the same thing at all: it does not ensure
that myrequirement is at least attempted before mydaemon runs. Instead,
it conditions the readiness of mydaemon to that of myrequirement. So,
it is "better" in the sense that it does not control another service
from a run script, but it is even further from what the OP wants.

  Any reference to another service in a  run script is going to be quirky
at best. Managing all kinds of dependencies between services is really
best done *outside* of run scripts, which is why s6-rc exists. It does
not currently have all the expressive power of systemd for dependencies,
but in the future, it will.

--
  Laurent

Re: S6 Queries

[Arjun D R]

[-- Attachment #1: Type: text/plain, Size: 5682 bytes --]

Thanks Laurent for the detailed explanations. We did a bootup speed
comparison between S6 and systemd. S6 is able to boot up slightly faster
than systemd. Actual result is 4-4.5% faster but we were expecting
something near to 20%.
Ours is a bit complex setup with more than 140 services (includes a lot of
long run services and a lot of dependencies). The main advantage in systemd
is, it starts many critical processes very quickly since it has no
dependency to logging services. We collect the logs from journalctl and
store it in log files. Whereas in S6, the critical services start up is a
bit delayed since it has to depend on logging services which in turn
depends on other services (responsible for backing up the previous logs).

Arjun

On Mon, Aug 2, 2021 at 1:57 PM Laurent Bercot <ska-supervision@skarnet.org>
wrote:

> >1. In systemd, the services are grouped as targets and each target depends
> >on another target as well. They start as targets. [ex: Reached
> >local-fs.target, Reached network.target, Reached UI target,...]. Is there
> >any way in S6 to start the init system based on bundles?
>
>   Yes, that is what bundles are for. In your stage 2 boot script
> (typically /etc/s6-linux-init/current/scripts/rc.init), you should
> invoke s6-rc as:
>    s6-rc change top
> if "top" is the name of your default bundle, i.e. the bundle that
> contains all the services you want to start at boot time. You can
> basically convert the contents of your systemd targets directly into
> contents of your s6-rc bundles; and you decide which one will be
> brought up at boot time via the s6-rc invocation in your stage 2
> init script.
>
>
> >2. Are there any ways to have loosely coupling dependencies? In systemd,
> we
> >have After=. After option will help the current service to start after the
> >mentioned service (in after). And the current service will anyway start
> >even if the mentioned service in After fails to start. Do we have such
> >loosely coupled dependency facility in S6?
>
>   Not at the moment, no. The next version of s6-rc will allow more types
> of dependencies, with clearer semantics than the systemd ones (After=,
> Requires= and Wants= are not orthogonal, which is unintuitive and causes
> misuse); but it is still in early development.
>
>   For now, s6-rc only provides one type of dependency, which is the
> equivalent of Requires+After. I realize this is not flexible enough
> for a lot of real use cases, which is one of the reasons why another
> version is in development. :)
>
>
> >3. Is there any tool available in S6 to measure the time taken by each
> >service to start? We can manually measure it from the logs, but still
> >looking for a tool which can provide accurate data.
>
>   Honestly, if you use the -v2 option to your s6-rc invocation, as in
>    s6-rc -v2 change top
> and you ask the catch-all logger to timestamp its lines (which should
> be the default, but you can change the timestamp style via the -t
> option to s6-linux-init-maker)
> then the difference of timestamps between the lines:
>    s6-rc: info: service foo: starting
> and
>    s6-rc: info: service foo successfully started
> will give you a pretty accurate measurement of the time it took service
> foo to start. These lines are written by s6-rc exactly as the
> "starting" or "completed" event occurs, and they are timestamped by
> s6-log immediately; the code path is the same for both events, so the
> delays cancel out, and the only inaccuracy left is randomness due to
> scheduling, which should not be statistically significant.
>
>   At the moment, the s6-rc log is the easiest place to get this data
> from. You could probably hack something with the "time" shell command
> and s6-svwait, such as
>    s6-svwait -u /run/service/foo ; time s6-svwait -U /run/service/foo
> which would give you the time it took for foo to become ready; but
> I doubt it would be any more accurate than using the timestamps in the
> s6-rc logs, and it's really not convenient to set up.
>
>
> >4. Does the S6 init system provide better boot up performance compared to
> >systemd ?  One of our main motives is to attain better bootup performance.
> >Is our expectation correct?
>
>   The boot up performance should be more or less *similar* to systemd.
> The code paths used by the s6 ecosystem are much shorter than the ones
> used by systemd, so _in theory_ you should get faster boots with s6.
>
>   However, systemd cheats, by starting services before their dependencies
> are ready. For instance, it will start services before any kind of
> logging is ready, which is pretty dangerous for several reasons. As a
> part of its "socket activation" thing, it will also claim readiness
> on some sockets before even attempting to run the actual serving
> processes (which may fail, in which case readiness was a lie.)
> Because of that, when everything goes well, systemd cuts corners that
> s6 does not, and may gain some advantage.
>
>   So all in all, I expect that depending on your system, the difference
> in speed will not be remarkable. On very simple setups (just a few
> services), systemd's overhead may be noticeable and you may see real
> improvements with s6. On complex setups with lots of dependencies, s6
> might still have the speed advantage but I don't think it will be
> anything amazing. The real benefit of s6 is that it achieves
> roughly the same speed as systemd *while being more reliable and
> predictable*.
>
>   If you actually make bootup speed comparisons between systemd and s6,
> please share them! I am interested in that kind of benchmarks, and
> I'm sure the community would like to see numbers as well.
>
> --
>   Laurent
>
>

Re: S6 Queries

[Laurent Bercot]

>Thanks Laurent for the detailed explanations. We did a bootup speed
>comparison between S6 and systemd. S6 is able to boot up slightly faster
>than systemd. Actual result is 4-4.5% faster but we were expecting
>something near to 20%.
>Ours is a bit complex setup with more than 140 services (includes a lot of
>long run services and a lot of dependencies). The main advantage in systemd
>is, it starts many critical processes very quickly since it has no
>dependency to logging services. We collect the logs from journalctl and
>store it in log files. Whereas in S6, the critical services start up is a
>bit delayed since it has to depend on logging services which in turn
>depends on other services (responsible for backing up the previous logs).

  Thank you for these numbers! Indeed they confirm my intuition: booting
via s6 is a little faster than systemd, but nothing extraordinary,
because s6-rc emphasizes reliability over speed.

  Your critical services may be slightly delayed, but you have the
guarantee that you will not lose logs - whereas with systemd, if a piece
of your logging system fails to start, your critical services may 
already
have produced some data that will vanish into the aether. Whether or not
that's an acceptable risk is up to you.

--
  Laurent