supervision - discussion about system services, daemon supervision, init, runlevel management, and tools such as s6 and runit
 help / color / mirror / Atom feed
* Pattern for multiple subservices and dynamic discovery i.e. VPN
@ 2022-08-17  8:32 Oliver Schad
  2022-08-17 11:04 ` Laurent Bercot
  0 siblings, 1 reply; 8+ messages in thread
From: Oliver Schad @ 2022-08-17  8:32 UTC (permalink / raw)
  To: supervision

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Hello everybody,

I'm looking for a pattern to solve a problem, where you have to
discover dynamically the services you have to start.

Examples could be VPN configurations, where you discover the
configuration files and start for every file an instance of the VPN
service.

But it would be nice to have supervision for all of these instances
instead of doing it without. Of course I know how to do it without.

Is there a pattern around to solve this with s6 and supervision?

Best Regards
Oli

-- 
Automatic-Server AG •••••
Oliver Schad
Geschäftsführer
Hardstr. 46
9434 Au | Schweiz

www.automatic-server.com | oliver.schad@automatic-server.com
Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-17  8:32 Pattern for multiple subservices and dynamic discovery i.e. VPN Oliver Schad
@ 2022-08-17 11:04 ` Laurent Bercot
  2022-08-18  9:32   ` Oliver Schad
  0 siblings, 1 reply; 8+ messages in thread
From: Laurent Bercot @ 2022-08-17 11:04 UTC (permalink / raw)
  To: supervision

>
>I'm looking for a pattern to solve a problem, where you have to
>discover dynamically the services you have to start.
>
>Examples could be VPN configurations, where you discover the
>configuration files and start for every file an instance of the VPN
>service.

  Hi Oliver,

  Dynamic instantiation is a real pain point - it's an often requested
feature, but it's surprisingly hard to make it work correctly and
safely in a supervision scheme. Supervision works very well in static
environments, but dynamic discovery is at odds with the architecture.

  I have a few ideas to mitigate that and help people create instanced
services. Instantiation is a planned feature of the future s6-rc v1
but it's still a ways away; I am also thinking of adding tools to help
people handle instances with regular s6, amd they may come in the near
future, but there are currently no such helpers, sorry.

--
  Laurent


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-17 11:04 ` Laurent Bercot
@ 2022-08-18  9:32   ` Oliver Schad
  2022-08-18 10:04     ` Davor Ocelic
  2022-08-18 11:40     ` Laurent Bercot
  0 siblings, 2 replies; 8+ messages in thread
From: Oliver Schad @ 2022-08-18  9:32 UTC (permalink / raw)
  To: supervision; +Cc: Laurent Bercot

[-- Attachment #1: Type: text/plain, Size: 2060 bytes --]

Dear Laurent,

thanks a lot for your quick response. I understand, that dynamic
discovery needs some thoughts to implement in an easy and flexible way.

Just to understand the requirements, if we want to implement that today:

- we need an scanning component for the desired state of running
  instances (something like 'find /etc/openvpn -name "*conf"')
- we need an scanning component for the current state in process list
- we need a diffing component
- we need a state applier component

A component could be a shell function. Applying state would mean to
fork and exec. A process tree would look like

/usr/bin/s6-svscan
 s6-supervise vpn-manager
   vpn-manager
     openvpn foo1.conf
     openvpn foo2.conf

Best Regards
Oli

On Wed, 17 Aug 2022 11:04:50 +0000
"Laurent Bercot" <ska-supervision@skarnet.org> wrote:

> >
> >I'm looking for a pattern to solve a problem, where you have to
> >discover dynamically the services you have to start.
> >
> >Examples could be VPN configurations, where you discover the
> >configuration files and start for every file an instance of the VPN
> >service.  
> 
>   Hi Oliver,
> 
>   Dynamic instantiation is a real pain point - it's an often requested
> feature, but it's surprisingly hard to make it work correctly and
> safely in a supervision scheme. Supervision works very well in static
> environments, but dynamic discovery is at odds with the architecture.
> 
>   I have a few ideas to mitigate that and help people create instanced
> services. Instantiation is a planned feature of the future s6-rc v1
> but it's still a ways away; I am also thinking of adding tools to help
> people handle instances with regular s6, amd they may come in the near
> future, but there are currently no such helpers, sorry.
> 
> --
>   Laurent
> 



-- 
Automatic-Server AG •••••
Oliver Schad
Geschäftsführer
Hardstr. 46
9434 Au | Schweiz

www.automatic-server.com | oliver.schad@automatic-server.com
Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-18  9:32   ` Oliver Schad
@ 2022-08-18 10:04     ` Davor Ocelic
  2022-08-18 12:39       ` Oliver Schad
  2022-08-18 11:40     ` Laurent Bercot
  1 sibling, 1 reply; 8+ messages in thread
From: Davor Ocelic @ 2022-08-18 10:04 UTC (permalink / raw)
  To: supervision

On Thu, Aug 18, 2022 at 11:32:30AM +0200, Oliver Schad wrote:

Hey,

How about an approach that would be more tightly integrated
with s6? For example:

> - we need an scanning component for the desired state of running
>   instances (something like 'find /etc/openvpn -name "*conf"')

Right, the scanning component would be the key part. Ideally it
would use inotify so that it doesn't have to poll and also it would
get the type of event automatically (file created or file deleted).

After it sees that a change in a directory has occurred (let's say a
file was added), then it does the following:

- It creates the appropriate service directory for this particular
  VPN config (possibly based on a template)
- It calls s6-svlink or the like to register the new service, which
  automatically also notifies s6-svscan that a new service has been
  added

> - we need an scanning component for the current state in process list
> - we need a diffing component
> - we need a state applier component

I hope these steps would not be necessary, since each .conf file would
be represented as a proper service, and would be managed by s6 natively.

Later, if/when a conf file is removed, the vpn-manager component would
do the opposite and call s6-svunlink.

As a final safeguard, in the `finish` script for such a service, the
script would check whether its original VPN config file is still present.
If it is not, it would trigger a permanent failure event for the service.

> A process tree would look like

/usr/bin/s6-svscan
 s6-supervise vpn-manager
 s6-supervise openvpn foo1.conf
 s6-supervise openvpn foo2.conf

(The individual openvpn services could also have a dependency on vpn-manager.)

Thanks,
Best regards,
Davor

> On Wed, 17 Aug 2022 11:04:50 +0000
> "Laurent Bercot" <ska-supervision@skarnet.org> wrote:
> 
> > >
> > >I'm looking for a pattern to solve a problem, where you have to
> > >discover dynamically the services you have to start.
> > >
> > >Examples could be VPN configurations, where you discover the
> > >configuration files and start for every file an instance of the VPN
> > >service.  
> > 
> >   Hi Oliver,
> > 
> >   Dynamic instantiation is a real pain point - it's an often requested
> > feature, but it's surprisingly hard to make it work correctly and
> > safely in a supervision scheme. Supervision works very well in static
> > environments, but dynamic discovery is at odds with the architecture.
> > 
> >   I have a few ideas to mitigate that and help people create instanced
> > services. Instantiation is a planned feature of the future s6-rc v1
> > but it's still a ways away; I am also thinking of adding tools to help
> > people handle instances with regular s6, amd they may come in the near
> > future, but there are currently no such helpers, sorry.
> > 
> > --
> >   Laurent
> > 
> 
> 
> 
> -- 
> Automatic-Server AG •••••
> Oliver Schad
> Geschäftsführer
> Hardstr. 46
> 9434 Au | Schweiz
> 
> www.automatic-server.com | oliver.schad@automatic-server.com
> Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47



^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-18  9:32   ` Oliver Schad
  2022-08-18 10:04     ` Davor Ocelic
@ 2022-08-18 11:40     ` Laurent Bercot
  1 sibling, 0 replies; 8+ messages in thread
From: Laurent Bercot @ 2022-08-18 11:40 UTC (permalink / raw)
  To: supervision

>- we need an scanning component for the desired state of running
>  instances (something like 'find /etc/openvpn -name "*conf"')
>- we need an scanning component for the current state in process list
>- we need a diffing component
>- we need a state applier component

  That sounds very much like what is planned for s6-rc v1, so I think
you will like it when it drops - but it won't be in the near future.

  However, I have some ideas for new s6 tools that wouldn't follow this
model directly but would make it easy for users to create and delete
new instance models, and add/remove instances - so your components
could be implemented over these tools by simple shell scripts. I'll
try to work on that soon.

--
  Laurent


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-18 10:04     ` Davor Ocelic
@ 2022-08-18 12:39       ` Oliver Schad
  2022-08-18 16:36         ` Laurent Bercot
  2022-08-18 18:18         ` Davor Ocelic
  0 siblings, 2 replies; 8+ messages in thread
From: Oliver Schad @ 2022-08-18 12:39 UTC (permalink / raw)
  To: supervision

[-- Attachment #1: Type: text/plain, Size: 1915 bytes --]

On Thu, 18 Aug 2022 06:04:25 -0400
Davor Ocelic <docelic-skarnet@spinlocksolutions.com> wrote:

> On Thu, Aug 18, 2022 at 11:32:30AM +0200, Oliver Schad wrote:
> How about an approach that would be more tightly integrated
> with s6? For example:
> 
> > - we need an scanning component for the desired state of running
> >   instances (something like 'find /etc/openvpn -name "*conf"')  
> 
> Right, the scanning component would be the key part. Ideally it
> would use inotify so that it doesn't have to poll and also it would
> get the type of event automatically (file created or file deleted).
> 
> After it sees that a change in a directory has occurred (let's say a
> file was added), then it does the following:
> 
> - It creates the appropriate service directory for this particular
>   VPN config (possibly based on a template)
> - It calls s6-svlink or the like to register the new service, which
>   automatically also notifies s6-svscan that a new service has been
>   added

That would just move 3 components to another level but they are
still needed: scanning existing service directories, diffing between
desired and current state and applying - so creating or removing
directories.

I see there a problem with multiple dynamic services. I'm not sure
about concurrency behaviour of updating processes in the service
directory. Maybe Laurent can explain problems in that area, if they
exist.

I'm not sure how complex the supervision itself is - however I would
love to solve the problem without doing supervision on my own. I
thought about your approach as well but it really depends how resilient
an update process is.

Best Regards
Oli

-- 
Automatic-Server AG •••••
Oliver Schad
Geschäftsführer
Hardstr. 46
9434 Au | Schweiz

www.automatic-server.com | oliver.schad@automatic-server.com
Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-18 12:39       ` Oliver Schad
@ 2022-08-18 16:36         ` Laurent Bercot
  2022-08-18 18:18         ` Davor Ocelic
  1 sibling, 0 replies; 8+ messages in thread
From: Laurent Bercot @ 2022-08-18 16:36 UTC (permalink / raw)
  To: supervision


>That would just move 3 components to another level but they are
>still needed: scanning existing service directories, diffing between
>desired and current state and applying - so creating or removing
>directories.

  So, diffing between desired and current state, and applying the
modifications are components of a *service manager*, not a supervision
suite, and it is important to maintain the distinction in order to
avoid scope creep in s6.

  Even when a service is *not* instanced, these components are somewhat
needed; it's just not noticed because their implementation over a
single supervised service is trivial. But it is important to remember
that the job of a supervision suite is to maintain the service in its
current state (up or down), *not* to manage the wanted state or apply
it. (Of course, it does provide tools to perform state transitions
for longruns, but it comes with no policy on when to call these tools.)

  The components you want definitely have their place in s6-rc; but in
the meantime, they can also be scripted on top of regular s6 if you
have a good modelization for implementing instances, which I will add
in the near future.


>I see there a problem with multiple dynamic services. I'm not sure
>about concurrency behaviour of updating processes in the service
>directory. Maybe Laurent can explain problems in that area, if they
>exist.

  s6 manages processes and every supervised process needs its own
service directory. There will be as many service directories as
they are instances. (Some components of a template service directory
can of course be reused.) So there's no concurrency issue; however,
the instance management tool I'm thinking of could adopt various
updating methods depending on what you want. Best effort? Clean
shutdown, service replacement, then firing up of the new service's
instances? Rolling upgrade across the instances? These policies all
have their uses.


>I'm not sure how complex the supervision itself is - however I would
>love to solve the problem without doing supervision on my own. I
>thought about your approach as well but it really depends how resilient
>an update process is.

  It will definitely be resilient, but there are several ways to 
implement
it, see above.

--
  Laurent


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
  2022-08-18 12:39       ` Oliver Schad
  2022-08-18 16:36         ` Laurent Bercot
@ 2022-08-18 18:18         ` Davor Ocelic
  1 sibling, 0 replies; 8+ messages in thread
From: Davor Ocelic @ 2022-08-18 18:18 UTC (permalink / raw)
  To: supervision

On Thu, Aug 18, 2022 at 02:39:36PM +0200, Oliver Schad wrote:

> > - It creates the appropriate service directory for this particular
> >   VPN config (possibly based on a template)
> > - It calls s6-svlink or the like to register the new service, which
> >   automatically also notifies s6-svscan that a new service has been
> >   added
> 
> That would just move 3 components to another level but they are
> still needed: scanning existing service directories, diffing between
> desired and current state and applying - so creating or removing
> directories.

Right. And unfortunately that scanning/comparing of current and desired
states can quickly destroy the elegance of the original idea.

But I believe there is a way to make this work without explicitly scanning
(except for the necessary minimum) and without comparing or keeping internal
state. Various features and behavior details that Laurent already added to
s6 make this very doable.

Please take a look at: https://github.com/docelic/s6-dyn

It is a working demo I put together in the last couple hours to demonstrate
the proposed model.

The README covers a runnable demo / test case. After that, you could
read the comments in `manager/manager.rb` to get a better description of
the design decisions and alternative behaviors that are possible.

(The manager is in Ruby since I saw you using it. It could be any laguage.)

If you, Laurent, or the rest of the list have any comments or suggestions
for improvements, please share. (The manager is slightly customized
to your VPN example, but it can easily be extended and made generic.)

Thanks,
Davor

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2022-08-18 18:18 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-08-17  8:32 Pattern for multiple subservices and dynamic discovery i.e. VPN Oliver Schad
2022-08-17 11:04 ` Laurent Bercot
2022-08-18  9:32   ` Oliver Schad
2022-08-18 10:04     ` Davor Ocelic
2022-08-18 12:39       ` Oliver Schad
2022-08-18 16:36         ` Laurent Bercot
2022-08-18 18:18         ` Davor Ocelic
2022-08-18 11:40     ` Laurent Bercot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).