s6-rc user experience question

s6-rc user experience question

[Ihor Antonov]

Hi Laurent, hi list.

I've been trying to get s6-rc set up and after some time I think I
managed it. I want to describe my process as a list of steps and then
ask you for comments. So here it goes:

1. First we create 'scandir1', and put services there. Each service is a
   svcdir. We put dependencies file and type file in each svcdir.
   (We do not run svcscn on it, because it doesn't really manage
   dependencies)

2. We run s6-rc-compile pointing at scandir and get 'compiled' dir as
   output.

3. We run svcscan process on an emtpy dir - 'scandir2'

4. We run s6-rc-init , feeding 'compiled' and 'scandir2' dirs  we
   get 'live' dir.

At this point things seem to be working and I can use s6-rc to bring up
and down services with dependencies. But this gets very confusing and
does not look like a good user experience:

- The contents of scandir1 and scandir2 look very
similar (scandir2 differs only by the fact that it has symlinks instead
of regular svcdirs).

- Complied dir contains copies of scandir1 services plus some files

- Live dir is the most wired one. It has symlinks to compiled and
  scandir2 and it also has copies of services under servicedirs/
  directory

So same information is duplicated 3 times and symlinked 3 times.
Is this the intended flow? Or have I messed something up really badly?


Thanks

Ihor

Re: s6-rc user experience question

[Laurent Bercot]

>1. First we create 'scandir1', and put services there. Each service is a
>   svcdir. We put dependencies file and type file in each svcdir.
>   (We do not run svcscn on it, because it doesn't really manage
>   dependencies)

  That's a bad reason. :)
  The real reason why you don't run s6-svscan on a source s6-rc directory
is because it's not a suitable scan directory. It doesn't contain
service directories: it contains s6-rc definition directories, which
may be for longruns (and in this case *look like* service directories
but they're not, for instance the "run" file doesn't have to be
executable), but also for oneshots or bundles, and s6-svscan would not
know how to deal with those. And they have dependency information,
which s6-svscan cannot interpret, etc.

  Even if their formats are similar, a source directory and a scan
directory are not the same and aren't used for the same purposes.


>2. We run s6-rc-compile pointing at scandir and get 'compiled' dir as
>   output.
>
>3. We run svcscan process on an emtpy dir - 'scandir2'
>
>4. We run s6-rc-init , feeding 'compiled' and 'scandir2' dirs  we
>   get 'live' dir.
>
>At this point things seem to be working and I can use s6-rc to bring up
>and down services with dependencies.

  That is indeed the correct way to proceed.


>  But this gets very confusing and
>does not look like a good user experience:

  I agree it's confusing, and one of the reasons (but not the main one)
why s6-rc needs a redesign: you currently get access to all the internal
workings of the service manager, even when they're not relevant to you,
which is not ideal.


>So same information is duplicated 3 times and symlinked 3 times.
>Is this the intended flow? Or have I messed something up really badly?

  It's the intended flow, and the information duplication is necessary.

  - The source directories are where you, the user, edit the information.
It's your working copy of it. You should be able to change it as you
wish; the modifications you make should not impact the system.

  - The compiled directory is where the information is stored when
s6-rc-compile validates your source directories. It's an automation-
approved snapshot of the state of your source directories at the time
you called s6-rc-compile. It is immutable: neither you, nor the system
can modify it - and you can store it on a read-only filesystem. The
point is that once you have it, and it works once, it will *always* 
work.

  - The live directory is the working copy of the *system*. It manages
what is going on *right now* on your machine. Some of it is symbolic
links to a compiled directory, when the system only needs to read
information; but some of it is state that can be changed by the system
and that needs to be a copy. That includes longrun service directories,
that maintain running state, which is why these need to be copied
from the compiled directory.

  Having this duplication is the only way of ensuring that your
modifications do not change the snapshots you take when compiling and
do not impact the current machine state, and also that the operation
of your current services does not impact the set of services you're
booting on (which could lead to failed boots).

  Other service managers do not make the distinction between the
user working set, the system working set, and immutable snapshots; that
is a big issue I have with them, because it's difficult to make them
safe.

  Hope this helps,

--
  Laurent

Re: s6-rc user experience question

[Ihor Antonov]

On 2022-10-16 10:39, Laurent Bercot wrote:
>  ...
>  I agree it's confusing, and one of the reasons (but not the main one)
> why s6-rc needs a redesign: you currently get access to all the internal
> workings of the service manager, even when they're not relevant to you,
> which is not ideal.
 
Perhaps I can offer a few suggestions how to improve usability:
  - combine compile + svscan on empty dir + init steps into one, like
    `s6-rc init source_dir` and it does those steps under the hood.

  - maybe instead of creating file based database take on sqlite as a 
    dependency and keep compiled + live state there? sqlite is
    ubiquitous and very light weight. It can save the trouble of
    inventing our own compiled database formats and folder strucutres


> ... 
>  Having this duplication is the only way of ensuring that your
> modifications do not change the snapshots you take when compiling and
> do not impact the current machine state, and also that the operation
> of your current services does not impact the set of services you're
> booting on (which could lead to failed boots).
> 
>  Other service managers do not make the distinction between the
> user working set, the system working set, and immutable snapshots; that
> is a big issue I have with them, because it's difficult to make them
> safe.

This makes sense. Popular cloud infrastructure management tool Terraform
does this too. It does 3-way diff on each operation, comparing desired
state of the system, currently observed state, and state recorded after
finishing previous operation. This way it knows when things need to be
added/removed/modified. This cannot be done reliably with 2-way diff
approach, and tools like ansible fall short here. Removal of a resource
from ansible playbook simply makes ansible forget about it, where
terraform detects the absense and undersands that previously managed
resource needs to be de-provisioned.

What you describe sounds very similar, but applied to the state of
service dependency tree.


>  Hope this helps,
It does, thank you. What are your plans / thoughts on s6-rc redesign?

Ihor

Re: s6-rc user experience question

[Laurent Bercot]

>Perhaps I can offer a few suggestions how to improve usability:
>  - combine compile + svscan on empty dir + init steps into one, like
>    `s6-rc init source_dir` and it does those steps under the hood.

  No, because these are operations that are ideally done at different
times.
  - Compilation happens "offline", before you (re)boot the machine.
  - s6-svscan is run independently from s6-rc, typically very early
during the boot sequence. (Ideally it's process 1.) In any case the
preexistence of a supervision tree is assumed by s6-rc, so spawning
one as part of an s6-rc command isn't good workflow.
  - Initialization happens when the service manager is ready to run,
which means when the machine is online and already has an s6-svscan
running.

  Combining these operations is only useful if you're doing things in
a very convoluted, suboptimal way.


>  - maybe instead of creating file based database take on sqlite as a    dependency and keep compiled + live state there? sqlite is
>    ubiquitous and very light weight. It can save the trouble of
>    inventing our own compiled database formats and folder strucutres

  The format of the compiled database is entirely orthogonal to the
usability. Adding a dependency to an external package would do exactly
nothing to improve the UX here; it would just make the code bigger,
slower, more complex, harder to maintain and less secure. And that's
without mentioning that a relational database is not at all the right
structure for the kind of data that s6-rc deals with.

  Not a great trade-off, if you ask me.

  However, there's a change I can make that would immediately improve
s6-rc's usability: since it's largely a perception issue, changing
the vocabulary from "s6-rc-compile" to "s6-rc-snapshot" and
"compiled database" to "opaque snapshot of the set of services" would
be a good idea. It would spook fewer people, and send fewer people
into red herring trains of thoughts. :)


>What are your plans / thoughts on s6-rc redesign?

  It's ongoing. It's difficult, and I always find a myriad of things to
do to distract me from working on it, so it's going (very) slowly. But
it's always lurking in the back of my mind, and there isn't a day that
I don't wonder what I have gotten myself into.

  The initial call for funds is still readable here:
  https://skarnet.com/projects/service-manager.html

--
  Laurent

Re: s6-rc user experience question

[Ihor Antonov]

On 2022-10-17 13:11, Laurent Bercot wrote:
> 
>  No, because these are operations that are ideally done at different
> times.
>  - Compilation happens "offline", before you (re)boot the machine.
>  - s6-svscan is run independently from s6-rc, typically very early
> during the boot sequence. (Ideally it's process 1.) In any case the
> preexistence of a supervision tree is assumed by s6-rc, so spawning
> one as part of an s6-rc command isn't good workflow.
>  - Initialization happens when the service manager is ready to run,
> which means when the machine is online and already has an s6-svscan
> running.

I understand that these tools are executed at different times and we
want to make long-lived daemons as small as possible, that only do one
job. Perhaps a higher-level orchestration tool(s) is/are needed, that
will accomplish most typical workflows like:

- make changes in source -> complie -> switch symlink to latest compiled snapshot ->
  s6-rc-update. In systmd world this is easier: edit -> systemd
  daemon-reload

- list running services, start/stop them etc.

- auto-generating cooresponding logger service for a service. Today it
  is mostly a copy-paste job and incompatible with vanilla svcscan
  ./log/run subdir approach, which is confusing on its own.


Another set of reasons for a high-level utility is better UX:

- today users have to interact with multiple binaries and remember all
  their names, cli flags and positional parameters.

- there are no manual pages that would help with the point above, and
  short help messages are not very useful. ( I often resort to reading 
  https://skarnet.org/software/s6 from my browser )

- CLI interface of those binaries is not very intuitive, and there are
  no long options to improve readability in scripts. 
  Example: It is very hard to remember what this actually means
  
        s6-log -d3 -- t s16777216 n64 /var/log/mysvc

  I would much prefer something like this for all utilities, in
  addition to short options

        s6-log --notification-fd 3
               --num-files 64
               --filesize 16777216
               --timestamp tai64
                /var/log/mysvc


If changing CLI interface of s6 toolchest is not something that you
consider then a higher-level orchestration tool can take away some these
pains

>  However, there's a change I can make that would immediately improve
> s6-rc's usability: since it's largely a perception issue, changing
> the vocabulary from "s6-rc-compile" to "s6-rc-snapshot" and
> "compiled database" to "opaque snapshot of the set of services" would
> be a good idea. It would spook fewer people, and send fewer people
> into red herring trains of thoughts. :)

If compiled snapshot was an opaque file, instead of a directory that
would indeed make the perception a bit better

Re: s6-rc user experience question

[Laurent Bercot]

>  Perhaps a higher-level orchestration tool(s) is/are needed, that
>will accomplish most typical workflows like: (...)

  These are all valid points, and things that ultimately s6-frontend,
the future UI over s6/s6-linux-init/s6-rc, aims to solve.
"Higher-level interface" is the (now) #1 feature request for the s6
family of tools, so be assured I've heard it. :)

  No promises on the date of delivery, but it's definitely on the radar.

--
  Laurent

Re: s6-rc user experience question

[Alexis]

Ihor Antonov <ihor@antonovs.family> writes:

> - there are no manual pages that would help with the point 
> above, and
>   short help messages are not very useful. ( I often resort to 
>   reading 
>   https://skarnet.org/software/s6 from my browser )

i might well be misunderstanding what you're saying, but as 
Laurent notes on the s6 Web page, i've ported the docs for s6 (and 
for execline, s6-networking and s6-portable-utils) to mdoc(7) man 
pages:

   https://github.com/flexibeast/s6-man-pages

which at least some distros provide as a package.


Alexis.

Re: s6-rc user experience question

[Ihor Antonov]

On 2022-10-18 12:13, Alexis wrote:
>   https://github.com/flexibeast/s6-man-pages
> 
> which at least some distros provide as a package.
> 
> 
Hi, Alexis. Thanks for pointing out. I am on FreeBSD and it looks like
it is not packaged here. I will see if I can package it myslef, if time
permits.

Ihor