s6-supervise: use of nosetsid

s6-supervise: use of nosetsid

[Laurent Bercot]

  Hello,

  The next version of s6 will be a major bump, with a few long-awaited
QoL changes - mainly a thorough cleanup of how s6-svscan handles
signals and the various commands sent by s6-svscanctl, but also some
goodies that you should like. :)

  One issue that has been often reported by users is that when they
try running s6-svscan in a terminal, and then ^C to kill it, the
services remain running. This is intentional, because supervision
suites are designed to isolate processes from anything accidental that
could bring them down, and in particular services should normally
survive supervisor death - but so far there has been many more
instances of people having trouble with that behaviour than instances
of s6-supervise accidentally dying.

  I have previously added the "nosetsid" feature to s6-supervise, to
address the issue: having a "nosetsid" file in a service directory
prevents the service from being started as a session leader, it starts
in the same session as the supervision tree (and, if the nosetsid file
is empty, in the same process group). So when people want to manually
test a supervision tree, they can have nosetsid files in their test
service directories, and ^C will send a SIGINT to all the processes
including the services, so everything will die, which is what they
want.

  There are two problems with the nosetsid approach:

  - Oftentimes, users are not aware of the existence of nosetsid, and
still experience the issue. It's almost an s6 FAQ at this point.
  - The nosetsid functionality is inherently a risk: it puts the
whole supervision tree at the mercy of a misbehaved service that would
send a signal to its whole process group. There is a reason why
s6-supervise normally starts services in a different session, and
nosetsid bypasses that safety measure.

  So I am thinking of another approach to make s6 friendlier to users
who would - despite it not being recommended behaviour - test a
supervision tree in a terminal: have s6-supervise handle SIGINT and
make it kill its service before exiting. That would ensure that ^C
cleans up everything.

  This approach has the drawback of making services a little less
resilient, but s6-supervise getting a SIGINT should *only* happen in
the case of someone running a supervision tree in a terminal, which
is absolutely not something that should exist in production, so it's
probably not a big concern. However, it comes with a major advantage:
it removes the original reason for the addition of nosetsid.
  So, with the change to ^C handling, I am considering removing the
dangerous nosetsid functionality entirely.

  Hence, my question to users: do you have a *valid* reason to use
nosetsid files in your service directories? Are there use cases for
nosetsid that I have not thought about, and that would make using s6
impractical if the functionality were to be removed?

  Thanks in advance for your input.

--
  Laurent

Re: s6-supervise: use of nosetsid

[Casper Ti. Vector]

On Thu, Dec 03, 2020 at 04:46:58PM +0000, Laurent Bercot wrote:
>  Hence, my question to users: do you have a *valid* reason to use
> nosetsid files in your service directories? Are there use cases for
> nosetsid that I have not thought about, and that would make using s6
> impractical if the functionality were to be removed?

dcron crashes without it, which is why [1] exists.  The log says
`setpgid: Operation not permitted'.  I have not investigated further.

[1] <https://gitea.com/CasperVector/slew/src/branch/master/pkg/dcron/base/dcron/nosetsid>

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2022.09.20)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: s6-supervise: use of nosetsid

[Laurent Bercot]

>dcron crashes without it, which is why [1] exists.  The log says
>`setpgid: Operation not permitted'.  I have not investigated further.

  The problem happens here:
  https://github.com/dubiousjim/dcron/blob/master/main.c#L272

  dcron assumes that it's not a session leader when run in the 
foreground.
This is a programming error that should be reported and fixed; if
setpgid(0, 0) fails with EPERM, it just means that the process is
already a session leader, and the program should simply keep running
instead of exiting.

  So, not a valid reason. ;)

  As a workaround, you can avoid nosetsid by just running dcrond under
execline's 'trap', or a shell, or basically any program that will
just forward signals to it.

--
  Laurent

Re: s6-supervise: use of nosetsid

[Casper Ti. Vector]

On Thu, Dec 03, 2020 at 05:21:44PM +0000, Laurent Bercot wrote:
> https://github.com/dubiousjim/dcron/blob/master/main.c#L272
Thanks for the explanation; I will report it to the upstream.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2022.09.20)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: s6-supervise: use of nosetsid

[Steve Litt]

On Thu, 03 Dec 2020 16:46:58 +0000
"Laurent Bercot" <ska-supervision@skarnet.org> wrote:

>   Hello,
> 
>   The next version of s6 will be a major bump, with a few long-awaited
> QoL changes - mainly a thorough cleanup of how s6-svscan handles
> signals and the various commands sent by s6-svscanctl, but also some
> goodies that you should like. :)
> 
>   One issue that has been often reported by users is that when they
> try running s6-svscan in a terminal, and then ^C to kill it, the
> services remain running. This is intentional, because supervision
> suites are designed to isolate processes from anything accidental that
> could bring them down, and in particular services should normally
> survive supervisor death - but so far there has been many more
> instances of people having trouble with that behaviour than instances
> of s6-supervise accidentally dying.
> 
>   I have previously added the "nosetsid" feature to s6-supervise, to
> address the issue: having a "nosetsid" file in a service directory
> prevents the service from being started as a session leader, it starts
> in the same session as the supervision tree (and, if the nosetsid file
> is empty, in the same process group). So when people want to manually
> test a supervision tree, they can have nosetsid files in their test
> service directories, and ^C will send a SIGINT to all the processes
> including the services, so everything will die, which is what they
> want.
> 
>   There are two problems with the nosetsid approach:
> 
>   - Oftentimes, users are not aware of the existence of nosetsid, and
> still experience the issue. It's almost an s6 FAQ at this point.
>   - The nosetsid functionality is inherently a risk: it puts the
> whole supervision tree at the mercy of a misbehaved service that would
> send a signal to its whole process group. There is a reason why
> s6-supervise normally starts services in a different session, and
> nosetsid bypasses that safety measure.
> 
>   So I am thinking of another approach to make s6 friendlier to users
> who would - despite it not being recommended behaviour - test a
> supervision tree in a terminal: have s6-supervise handle SIGINT and
> make it kill its service before exiting. That would ensure that ^C
> cleans up everything.
> 
>   This approach has the drawback of making services a little less
> resilient, but s6-supervise getting a SIGINT should *only* happen in
> the case of someone running a supervision tree in a terminal, which
> is absolutely not something that should exist in production, so it's
> probably not a big concern. However, it comes with a major advantage:
> it removes the original reason for the addition of nosetsid.
>   So, with the change to ^C handling, I am considering removing the
> dangerous nosetsid functionality entirely.
> 
>   Hence, my question to users: do you have a *valid* reason to use
> nosetsid files in your service directories? Are there use cases for
> nosetsid that I have not thought about, and that would make using s6
> impractical if the functionality were to be removed?
> 
>   Thanks in advance for your input.

I have no reason for nosetsid files.

I'm a big fan of keeping things the same and keeping them simple. What
happens, in the current s6, if somebody first Ctrl+C's s6-svscan from
the terminal, and then issues a pkill s6-supervise, with whatever
signal will kill a s6-supervise instance?

If the actual daemons survive the death of their individual s6-supervise
supervisors after the pkill, then yes, you could modify s6-supervise to
kill the daemon they're supervising. You could even make it an option by
having a certain filename turn that behavior off, if people want that.

SteveT

Steve Litt 
Autumn 2020 featured book: Thriving in Tough Times
http://www.troubleshooters.com/thrive

Re: s6-supervise: use of nosetsid

[Guillermo]

El jue, 3 dic 2020 a las 13:47, Laurent Bercot escribió:
>
>   I have previously added the "nosetsid" feature to s6-supervise, to
> address the issue: [...]. So when people want to manually
> test a supervision tree, they can have nosetsid files in their test
> service directories, and ^C will send a SIGINT to all the processes
> including the services, so everything will die, which is what they
> want.
> [...]
>   Hence, my question to users: do you have a *valid* reason to use
> nosetsid files in your service directories?

I don't know if it is a valid reason, so I'll just tell what I
remember using these files for, and you judge :)

I happen to be one of those people who test service directories and
s6-rc service definitions by running s6-svscan in a terminal using
's6-svscan &' (and then s6-rc-init with a live state directory that is
not /run/s6-rc, and lots of -l options), and tearing down the
supervision tree using 's6-svscanctl -t .'. So no Ctrl + C in my case.

I remember occasionally using 'nosetsid' files for the convenience of
being able to redirect the output of 'run' and 'finish' to /dev/tty
(the controlling terminal of the shell I used to run s6-svscan) so
that I could see certain messages during tests in which s6-svscan had
its stdout & stderr redirected to a logger (typically, for emulating
an s6-linux-init-like s6-svscan-as-process-1 scenario). I think it was
for doing something like

foreground {
  redirfd -w 1 /dev/tty
  echo "Some message I want displayed on my terminal"
}

Having the supervised processes become session leaders meant that they
lost the controlling terminal, which, for X terminals, would be some
random special file in the devpts filesystem. Redirecting to /dev/tty
would not work in that case.

Granted, it is a rather obscure usage, and not a frequent one in my
case, but your mentioning of notsetsid files triggered the memory, so
I thought I'd mention that.

G.

Re: s6-supervise: use of nosetsid

[Laurent Bercot]

>I remember occasionally using 'nosetsid' files for the convenience of
>being able to redirect the output of 'run' and 'finish' to /dev/tty
>(the controlling terminal of the shell I used to run s6-svscan) so
>that I could see certain messages during tests in which s6-svscan had
>its stdout & stderr redirected to a logger (typically, for emulating
>an s6-linux-init-like s6-svscan-as-process-1 scenario). I think it was
>for doing something like
>
>foreground {
>   redirfd -w 1 /dev/tty
>   echo "Some message I want displayed on my terminal"
>}

  Interesting. Would you be able to do the same thing with saving
/dev/tty as an extra file descriptor instead?

s6-svscan 3>&1 &

...

foreground {
   fdmove 1 3
   echo "Some message I want displayed on my terminal"
}

  Granted, this leaks a fd into all your services, but if it's for
terminal testing it shouldn't matter much. Otherwise, for maximum
hacking value, you could use the -X option to s6-svscan, which would
not leak the tty fd into all the services, but only transmit it as
s6-svscan-log's stderr - so in order to print something to the
terminal you'd have to send it to the catch-all logger, and configure
that logger with a filter that sends those messages to stderr. :)

  So, can you juggle fds to avoid nosetsid, or can't you do the same
thing without it?

--
  Laurent

Re: s6-supervise: use of nosetsid

[Guillermo]

El jue, 3 dic 2020 a las 18:34, Laurent Bercot escribió:
>
>   So, can you juggle fds to avoid nosetsid, or can't you do the same
> thing without it?

This particular use of nosetsid can probably be implemented in some
other way. I did think about having supervision tree processes inherit
a FD for the shell's terminal from s6-svscan, and also about using an
s6-fdholderd process, and having supervision tree processes retrieve
the FD from it. Although I haven't tested either yet.

It seems I missed the addition of s6-svscan's -X option; I saw it one
day in the output of 'ps' on a VM that was using s6-linux-init, didn't
recognize it, but then forgot to look up what it did :P But I would
have to 'tag' messages in some way, so that the catch-all logger
recognizes them and sends them to the terminal, right?

Anyway, nosetsid + /dev/tty is still the simplest implementation, I
have to admit.

G.

Re: s6-supervise: use of nosetsid

[Laurent Bercot]

>It seems I missed the addition of s6-svscan's -X option; I saw it one
>day in the output of 'ps' on a VM that was using s6-linux-init, didn't
>recognize it, but then forgot to look up what it did :P But I would
>have to 'tag' messages in some way, so that the catch-all logger
>recognizes them and sends them to the terminal, right?

  Yes. That's why it's the "advanced" solution over just leaking a fd
for /dev/tty.


>  Anyway, nosetsid + /dev/tty is still the simplest implementation, I
>have to admit.

  The more I think about it, the more I see any use of nosetsid as
incorrect. Implementing SIGINT handling for s6-supervise requires
jumping through serious hoops if the nosetsid function still exists, and
the point of my current work on s6 is to get things simpler, not more
complex.
  I'm leaning more and more in the direction of removing nosetsid, which
is why I want to help people find other ways to accomplish what they're
currently doing with it.

--
  Laurent

Re: s6-supervise: use of nosetsid

[eric vidal]

On Thu, 03 Dec 2020 16:46:58 +0000
"Laurent Bercot" <ska-supervision@skarnet.org> wrote:

> 
>   Hello,
> 
>   The next version of s6 will be a major bump, with a few long-awaited
> QoL changes - mainly a thorough cleanup of how s6-svscan handles
> signals and the various commands sent by s6-svscanctl, but also some
> goodies that you should like. :)
> 
>   One issue that has been often reported by users is that when they
> try running s6-svscan in a terminal, and then ^C to kill it, the
> services remain running. This is intentional, because supervision
> suites are designed to isolate processes from anything accidental that
> could bring them down, and in particular services should normally
> survive supervisor death - but so far there has been many more
> instances of people having trouble with that behaviour than instances
> of s6-supervise accidentally dying.
> 
>   I have previously added the "nosetsid" feature to s6-supervise, to
> address the issue: having a "nosetsid" file in a service directory
> prevents the service from being started as a session leader, it starts
> in the same session as the supervision tree (and, if the nosetsid file
> is empty, in the same process group). So when people want to manually
> test a supervision tree, they can have nosetsid files in their test
> service directories, and ^C will send a SIGINT to all the processes
> including the services, so everything will die, which is what they
> want.
> 
>   There are two problems with the nosetsid approach:
> 
>   - Oftentimes, users are not aware of the existence of nosetsid, and
> still experience the issue. It's almost an s6 FAQ at this point.
>   - The nosetsid functionality is inherently a risk: it puts the
> whole supervision tree at the mercy of a misbehaved service that would
> send a signal to its whole process group. There is a reason why
> s6-supervise normally starts services in a different session, and
> nosetsid bypasses that safety measure.
> 
>   So I am thinking of another approach to make s6 friendlier to users
> who would - despite it not being recommended behaviour - test a
> supervision tree in a terminal: have s6-supervise handle SIGINT and
> make it kill its service before exiting. That would ensure that ^C
> cleans up everything.
> 
>   This approach has the drawback of making services a little less
> resilient, but s6-supervise getting a SIGINT should *only* happen in
> the case of someone running a supervision tree in a terminal, which
> is absolutely not something that should exist in production, so it's
> probably not a big concern. However, it comes with a major advantage:
> it removes the original reason for the addition of nosetsid.
>   So, with the change to ^C handling, I am considering removing the
> dangerous nosetsid functionality entirely.
> 
>   Hence, my question to users: do you have a *valid* reason to use
> nosetsid files in your service directories? Are there use cases for
> nosetsid that I have not thought about, and that would make using s6
> impractical if the functionality were to be removed?
> 
>   Thanks in advance for your input.
> 
> --
>   Laurent
> 

I never used the nosetsid file.

Even for people who want to test a service with a fresh scandir from a terminal, i provide the necessary tools to properly start and stop the scandir without the need to hit the ^C key.

So no valid reason from my part to use this files.

Your commit https://github.com/skarnet/s6/commit/bdef68e12278ddfc5080732bd3b28dd5135c9d3a simplify a lot the s6-svscan interface which is a great thing.

-- 
eric vidal <eric@obarun.org>