Trouble starting services on boot with s6-rc

Trouble starting services on boot with s6-rc

[dark.pen9108]

[-- Attachment #1: Type: text/plain, Size: 4671 bytes --]

Hi,

Hoping this is the right place to ask for some help as I am very new to s6 and not well versed on any init system.  

I’m attempting to use what I understand to be the “new” s6-rc way to start some processes in a docker container. 

Just for testing out what I want to do, I have a oneshot who’s job is just to sleep, and a longrun that depends on the oneshot. 

I have created the following files 

[Aug 19 18:42:24 root@s6 ~]# ll /etc/s6-overlay/s6-rc.d/longrun_test/
total 12
-rw-rw-r-- 1 root root 13 Aug 17 23:19 dependencies
-rw-rw-r-- 1 root root 53 Aug 19 18:29 run
-rw-rw-r-- 1 root root  8 Aug 17 23:19 type
[Aug 19 18:42:24 root@s6 ~]# cat /etc/s6-overlay/s6-rc.d/longrun_test/type
longrun
[Aug 19 18:42:24 root@s6 ~]# cat /etc/s6-overlay/s6-rc.d/longrun_test/run
#!/command/execlineb -P
/usr/local/bin/longrun_start
[Aug 19 18:42:24 root@s6 ~]# cat /etc/s6-overlay/s6-rc.d/longrun_test/dependencies
sleeponstart
[Aug 19 18:42:24 root@s6 ~]# ll /etc/s6-overlay/s6-rc.d/sleeponstart/
total 8
-rw-rw-r-- 1 root root  8 Aug 15 21:53 type
-rw-rw-r-- 1 root root 52 Aug 16 20:25 up
[Aug 19 18:42:24 root@s6 ~]# cat /etc/s6-overlay/s6-rc.d/sleeponstart/type
oneshot
[Aug 19 18:42:24 root@s6 ~]# cat /etc/s6-overlay/s6-rc.d/sleeponstart/up
#!/command/execlineb -P
/usr/local/bin/sleeponstart

I can see that my services are registered with s6-rc 

[Aug 19 18:43:04 root@s6 ~]# s6-rc-db list all | grep -e longrun_test -e sleeponstart
sleeponstart
longrun_test

The test services are very simple scripts just to prove they ran
[Aug 19 18:44:23 root@s6 ~]# cat /usr/local/bin/sleeponstart
#!/bin/bash
sleep_time=30
sleep ${sleep_time}
echo "I slept for ${sleep_time}." > /tmp/i_slept
[Aug 19 18:44:23 root@s6 ~]# cat /usr/local/bin/longrun_start
#!/bin/bash
echo "I started the longrun" > /tmp/longrun_started

But when I look in tmp I never see the files created.

[Aug 19 18:44:23 root@s6 ~]# ll /tmp/
total 652
-rw------- 1 root root   5516 Mar  7 20:55 s6-overlay-noarch.tar.xz
-rw------- 1 root root 656864 Mar  7 20:55 s6-overlay-x86_64.tar.xz

If I start the longrun manually it and its dependency do run.  

[Aug 19 18:50:00 root@s6 ~]# s6-rc -a change longrun_test
[Aug 19 18:52:08 root@s6 ~]# ll /tmp/
total 660
-rw-r--r-- 1 root root     16 Aug 19 18:50 i_slept
-rw-r--r-- 1 root root     22 Aug 19 18:52 longrun_started
-rw------- 1 root root   5516 Mar  7 20:55 s6-overlay-noarch.tar.xz
-rw------- 1 root root 656864 Mar  7 20:55 s6-overlay-x86_64.tar.xz

So I would normally just think I am missing something to make them start on boot BUT, soon after boot I do see my oneshot running. 
[Aug 19 18:53:54 root@s6 /]# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.5  0.0    200    64 ?        Ss   18:53   0:00 /package/admin/s6/command/s6-svscan -d4 -- /run/service
root          14  0.0  0.0   4152  3104 pts/0    Ss+  18:53   0:00 /bin/sh -e /run/s6/basedir/scripts/rc.init top
root          15  0.0  0.0    204    64 ?        S    18:53   0:00 s6-supervise s6-linux-init-shutdownd
root          16  0.0  0.0    196     4 ?        Ss   18:53   0:00 /package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B
root          23  0.0  0.0    204    68 ?        S    18:53   0:00 s6-supervise s6rc-oneshot-runner
root          24  0.0  0.0    204    64 ?        S    18:53   0:00 s6-supervise longrun_test
root          25  0.0  0.0    204    60 ?        S    18:53   0:00 s6-supervise s6rc-fdholder
root          28  0.0  0.0    208    64 pts/0    S+   18:53   0:00 s6-rc -v2 -u -t 5000 -- change top
root          32  0.0  0.0    180     4 ?        Ss   18:53   0:00 /package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules
root          34  0.0  0.0    192    52 pts/0    S+   18:53   0:00 /package/admin/s6-2.11.1.0/command/s6-sudoc -e -t 30000 -T 4999 -- up 3
root          36  0.0  0.0    200    60 ?        S    18:53   0:00 /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --
root          38  0.0  0.0   4152  3080 ?        S    18:53   0:00 /bin/bash /usr/local/bin/sleeponstart <--------------
root          40  0.0  0.0   2616   940 ?        S    18:53   0:00 sleep 30 <-----------------
root          53  1.0  0.0   4812  3836 pts/1    Ss   18:53   0:00 /bin/bash
root          74  0.0  0.0   7396  3268 pts/1    R+   18:53   0:00 ps aux

Sorry if this was much more info than was relevant. Any help would be great. Thanx so much! 

-Benjamin

Re: Trouble starting services on boot with s6-rc

[Laurent Bercot]

>Hoping this is the right place to ask for some help as I am very new to s6 and not well versed on any init system.

  s6-overlay questions are normally asked in the "Issues" section of the
s6-overlay GitHub repository, but since yours are really s6-rc 
questions,
it's fine :)  (even though the answers are related to s6-overlay!)


>I can see that my services are registered with s6-rc
>[Aug 19 18:43:04 root@s6 ~]# s6-rc-db list all | grep -e longrun_test -e sleeponstart
>sleeponstart
>longrun_test

  That proves your services are *in the database*, not that they're going
to get started. To make sure your services are started on boot, you
should also check that they're declared in the "user" bundle (that's
the bundle where users declare what services are part of the final
machine state, by s6-overlay policy).

$ s6-rc-db contents user | grep -e longrun_test -e sleeponstart

  But since they *are* started when you boot your container, it means
they're indeed declared there, so that's not the issue. The real issue
is here:

>root          34  0.0  0.0    192    52 pts/0    S+   18:53   0:00 /package/admin/s6-2.11.1.0/command/s6-sudoc -e -t 30000 -T 4999 -- up 3

  The "-T 4999" part means there's a 5 second timeout on running your
sleeponstart service. And since this service is waiting for 30 seconds
before succeeding, and 30 > 5... it means that it times out after 5
seconds and gets killed, so the transition fails, and longrun_test
(which depends on it) is never brought up.

  s6-overlay-specific section:

  - You should see all that happening in your container's logs - s6-rc
prints informative messages when something fails. Unless you have
changed the S6_VERBOSITY value from the default 2 to something lower.

  - The 5 seconds timeout likely comes from the fact that you have not
modified the S6_CMD_WAIT_FOR_SERVICES_MAXTIME value, which is 5000 by
default. As is written in both the README[1] and the migration guide[2] 
:P
If your real service needs it, you can disable the timeout by adding to
your Dockerfile:
ENV S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
or you can adjust it to a bigger value than 5000.

  Good luck,

--
  Laurent


[1]: https://github.com/just-containers/s6-overlay/blob/master/README.md
[2]: 
https://github.com/just-containers/s6-overlay/blob/master/MOVING-TO-V3.md

Re: Trouble starting services on boot with s6-rc

[dark.pen9108]

[-- Attachment #1: Type: text/plain, Size: 4869 bytes --]

> 
>> root          34  0.0  0.0    192    52 pts/0    S+   18:53   0:00 /package/admin/s6-2.11.1.0/command/s6-sudoc -e -t 30000 -T 4999 -- up 3
> 
>   The "-T 4999" part means there's a 5 second timeout on running your
> sleeponstart service. And since this service is waiting for 30 seconds
> before succeeding, and 30 > 5... it means that it times out after 5
> seconds and gets killed, so the transition fails, and longrun_test
> (which depends on it) is never brought up.
> 
>   s6-overlay-specific section:
> 
>   - You should see all that happening in your container's logs - s6-rc
> prints informative messages when something fails. Unless you have
> changed the S6_VERBOSITY value from the default 2 to something lower.
> 
>   - The 5 seconds timeout likely comes from the fact that you have not
> modified the S6_CMD_WAIT_FOR_SERVICES_MAXTIME value, which is 5000 by
> default. As is written in both the README[1] and the migration guide[2] 
> :P
> If your real service needs it, you can disable the timeout by adding to
> your Dockerfile:
> ENV S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
> or you can adjust it to a bigger value than 5000.

ahh, very obvious. Sorry, I did read the docs(quite a few times) but starting from zero it was quite a lot to grok and I overlooked this(and I'm sure a number of other things). After adjusting S6_CMD_WAIT_FOR_SERVICES_MAXTIME, things are working as expected. 

I am a bit ashamed to admit I cannot find the logs. From reading https://wiki.gentoo.org/wiki/S6_and_s6-rc-based_init_system#logger I thought maybe I should be looking for file /run/uncaught-logs but could not find any such file in my docker instance(I understand, docker is not Gentoo).

While the docs did speak a lot to the directory structure used by s6, I still am finding it quite hard to figure out what the default directories are for some things. (e.g. I was clear on where my uncompiled s6-rc service directories should go but they seemed to "magically" get complied on boot and show up in a scan_dir) 

One additional item. As seems like not a great idea to smash the timeout for all services. Is there any way to adjust it on a per service basis? If not consider me a +1 to kindly add it to a wishlist somewhere.

Thanx so much for the quick reply, help, and great free software!

n.b. You should put up a BTC lightning "tip bucket" somewhere :)  

-Benjamin

On Sat, Aug 20, 2022, at 8:57 AM, Laurent Bercot wrote:
> >Hoping this is the right place to ask for some help as I am very new to s6 and not well versed on any init system.
> 
>   s6-overlay questions are normally asked in the "Issues" section of the
> s6-overlay GitHub repository, but since yours are really s6-rc 
> questions,
> it's fine :)  (even though the answers are related to s6-overlay!)
> 
> 
> >I can see that my services are registered with s6-rc
> >[Aug 19 18:43:04 root@s6 ~]# s6-rc-db list all | grep -e longrun_test -e sleeponstart
> >sleeponstart
> >longrun_test
> 
>   That proves your services are *in the database*, not that they're going
> to get started. To make sure your services are started on boot, you
> should also check that they're declared in the "user" bundle (that's
> the bundle where users declare what services are part of the final
> machine state, by s6-overlay policy).
> 
> $ s6-rc-db contents user | grep -e longrun_test -e sleeponstart
> 
>   But since they *are* started when you boot your container, it means
> they're indeed declared there, so that's not the issue. The real issue
> is here:
> 
> >root          34  0.0  0.0    192    52 pts/0    S+   18:53   0:00 /package/admin/s6-2.11.1.0/command/s6-sudoc -e -t 30000 -T 4999 -- up 3
> 
>   The "-T 4999" part means there's a 5 second timeout on running your
> sleeponstart service. And since this service is waiting for 30 seconds
> before succeeding, and 30 > 5... it means that it times out after 5
> seconds and gets killed, so the transition fails, and longrun_test
> (which depends on it) is never brought up.
> 
>   s6-overlay-specific section:
> 
>   - You should see all that happening in your container's logs - s6-rc
> prints informative messages when something fails. Unless you have
> changed the S6_VERBOSITY value from the default 2 to something lower.
> 
>   - The 5 seconds timeout likely comes from the fact that you have not
> modified the S6_CMD_WAIT_FOR_SERVICES_MAXTIME value, which is 5000 by
> default. As is written in both the README[1] and the migration guide[2] 
> :P
> If your real service needs it, you can disable the timeout by adding to
> your Dockerfile:
> ENV S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0
> or you can adjust it to a bigger value than 5000.
> 
>   Good luck,
> 
> --
>   Laurent
> 
> 
> [1]: https://github.com/just-containers/s6-overlay/blob/master/README.md
> [2]: 
> https://github.com/just-containers/s6-overlay/blob/master/MOVING-TO-V3.md
> 
> 

Re: Trouble starting services on boot with s6-rc

[Laurent Bercot]

>I am a bit ashamed to admit I cannot find the logs. From reading https://wiki.gentoo.org/wiki/S6_and_s6-rc-based_init_system#logger I thought maybe I should be looking for file /run/uncaught-logs but could not find any such file in my docker instance(I understand, docker is not Gentoo).

  By default, s6-overlay containers don't have a catch-all logger, so
all the logs fall through to the container's stdout/stderr.
  You can have an in-container catch-all logger that logs to
/run/uncaught-logs if you define S6_LOGGING to 1 or 2.


>While the docs did speak a lot to the directory structure used by s6, I still am finding it quite hard to figure out what the default directories are for some things. (e.g. I was clear on where my uncompiled s6-rc service directories should go but they seemed to "magically" get complied on boot and show up in a scan_dir)

  That's a thing with s6, and even s6-rc: it does not define policies, 
but
only mechanism. In other words: it lets you place things wherever you 
want,
there's no default.
  Of course, higher-level software that uses the s6 bricks needs to 
define
policies in order to get things done; that's why s6-overlay is a set of
scripts putting directories in certain places and calling s6 binaries
with certain arguments and options.
  s6-overlay uses s6-rc-compile under the hood, so it compiles your
database for you at boot time, to keep things as simple as possible 
(even
though it's not the optimal way of doing it).



>One additional item. As seems like not a great idea to smash the timeout for all services. Is there any way to adjust it on a per service basis? If not
>consider me a +1 to kindly add it to a wishlist somewhere.

  You can define timeout-up and timeout-down files in your s6-rc source
definition directories, they do just that. :)


>n.b. You should put up a BTC lightning "tip bucket" somewhere :)

  Thank you. Please see this Twitter thread:
  https://twitter.com/laurentbercot/status/1209247040657674240

--
  Laurent