Linuxisms in s6

Linuxisms in s6

[Jonathan de Boyne Pollard]

http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955 
, Adrian Chadd:

> We're using s6 at work, and it works out mostly ok. Mostly once you 
> get around the linuxisms, and the lack of sensible time code in it 
> (its calculations for daemon run duration is based on system time, not 
> wall clock, so if your box boots jan 1, 1970 then gets NTP, things 
> are.. hilarious), and some of the arcane bits to get logging working 
> right.
>
What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have 
Linuxisms, obviously.  But what Linuxisms does s6 have?

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Jan Bramkamp]

On 25/08/16 06:53, Jonathan de Boyne Pollard wrote:
> http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
> , Adrian Chadd:
>
>> We're using s6 at work, and it works out mostly ok. Mostly once you
>> get around the linuxisms, and the lack of sensible time code in it
>> (its calculations for daemon run duration is based on system time, not
>> wall clock, so if your box boots jan 1, 1970 then gets NTP, things
>> are.. hilarious), and some of the arcane bits to get logging working
>> right.
>>
> What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
> Linuxisms, obviously.  But what Linuxisms does s6 have?
>

The skalibs library used by s6 to calculate the deadlines should use 
clock_gettime(CLOCK_MONOTONIC) on FreeBSD and as such shouldn't be 
affected by changes to the wall clock.

I'm currently working on a FreeBSD only potential init replacement as 
well just without the mandatory per service supervisor process. The new 
kqueue EVFILT_PROCDESC filter type in FreeBSD 11 combined with pdfork() 
should make it really easy to deal child processes in a single unified 
kevent loop. Forking services could still be handled by a supervisor 
using procctl(PROC_REAP_ACQUIRE).

At the moment I'm fighting with some corner cases in the file descriptor 
passing code and redesigning the API to work without access to a 
writable file system. My last API required a writable file system 
because FreeBSD doesn't support listen()ing on unbound unix domain 
seqpacket sockets and I don't want to require something like the Linux 
/run tmpfs. Instead my new API uses socketpair() to create a connected 
pair of anonymous unix domain sockets for each supervised process. Next 
I have to find out if fexecve() works at least for fd 0, 1 and 2 without 
a mounted fdescfs.

I want to implement the following features in a single process capable 
of running as PID 1:
- Track service dependencies (want, require, bind, conflict)
- Store, Retrieve and close file descriptors.
- Spawn and supervise processes in a well defined environment.
- Reliable event notification with coalescing.
- Bootstrap the system with help from a default service.

With those features it should be able to wrap existing rc.d scripts 
without resorting to polling.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Adrian Chadd]

On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
<J.deBoynePollard-newsgroups@ntlworld.com> wrote:
> http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
> , Adrian Chadd:
>
>> We're using s6 at work, and it works out mostly ok. Mostly once you get
>> around the linuxisms, and the lack of sensible time code in it (its
>> calculations for daemon run duration is based on system time, not wall
>> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
>> hilarious), and some of the arcane bits to get logging working right.
>>
> What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
> Linuxisms, obviously.  But what Linuxisms does s6 have?

We just had a bunch of fun trying to get it to build right, and the
uptime stuff really threw us.

It's fine though, I found that s6 may start growing an IPC mechanism
so we could possibly do a launchd/jobd style service later (ie to run
things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
arbitrary messages, etc) so I may try incorporating it again. :)



-adrian
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Lars Engels]

[-- Attachment #1: Type: text/plain, Size: 1290 bytes --]

On Thu, Aug 25, 2016 at 08:46:10AM -0700, Adrian Chadd wrote:
> On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
> <J.deBoynePollard-newsgroups@ntlworld.com> wrote:
> > http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
> > , Adrian Chadd:
> >
> >> We're using s6 at work, and it works out mostly ok. Mostly once you get
> >> around the linuxisms, and the lack of sensible time code in it (its
> >> calculations for daemon run duration is based on system time, not wall
> >> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
> >> hilarious), and some of the arcane bits to get logging working right.
> >>
> > What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
> > Linuxisms, obviously.  But what Linuxisms does s6 have?
> 
> We just had a bunch of fun trying to get it to build right, and the
> uptime stuff really threw us.
> 
> It's fine though, I found that s6 may start growing an IPC mechanism
> so we could possibly do a launchd/jobd style service later (ie to run
> things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
> arbitrary messages, etc) so I may try incorporating it again. :)
> 

Can't this be done with devd? 

[-- Attachment #2: Type: application/pgp-signature, Size: 603 bytes --]

Re: Linuxisms in s6

[Adrian Chadd]

On 25 August 2016 at 12:48, Lars Engels <lars.engels@0x20.net> wrote:
> On Thu, Aug 25, 2016 at 08:46:10AM -0700, Adrian Chadd wrote:
>> On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
>> <J.deBoynePollard-newsgroups@ntlworld.com> wrote:
>> > http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
>> > , Adrian Chadd:
>> >
>> >> We're using s6 at work, and it works out mostly ok. Mostly once you get
>> >> around the linuxisms, and the lack of sensible time code in it (its
>> >> calculations for daemon run duration is based on system time, not wall
>> >> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
>> >> hilarious), and some of the arcane bits to get logging working right.
>> >>
>> > What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
>> > Linuxisms, obviously.  But what Linuxisms does s6 have?
>>
>> We just had a bunch of fun trying to get it to build right, and the
>> uptime stuff really threw us.
>>
>> It's fine though, I found that s6 may start growing an IPC mechanism
>> so we could possibly do a launchd/jobd style service later (ie to run
>> things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
>> arbitrary messages, etc) so I may try incorporating it again. :)
>>
>
> Can't this be done with devd?

Sure, but I'm looking for something more generic than just devd. Like,
notifications about events like "default route is up" can be done by
sniffing the rtsock, but notifications like "ntpdate has updated the
date, we can now do crypto services" doesn't happen there right now.



-adrian
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Warner Losh]

On Thu, Aug 25, 2016 at 3:08 PM, Adrian Chadd <adrian@freebsd.org> wrote:
> On 25 August 2016 at 12:48, Lars Engels <lars.engels@0x20.net> wrote:
>> On Thu, Aug 25, 2016 at 08:46:10AM -0700, Adrian Chadd wrote:
>>> On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
>>> <J.deBoynePollard-newsgroups@ntlworld.com> wrote:
>>> > http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
>>> > , Adrian Chadd:
>>> >
>>> >> We're using s6 at work, and it works out mostly ok. Mostly once you get
>>> >> around the linuxisms, and the lack of sensible time code in it (its
>>> >> calculations for daemon run duration is based on system time, not wall
>>> >> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
>>> >> hilarious), and some of the arcane bits to get logging working right.
>>> >>
>>> > What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
>>> > Linuxisms, obviously.  But what Linuxisms does s6 have?
>>>
>>> We just had a bunch of fun trying to get it to build right, and the
>>> uptime stuff really threw us.
>>>
>>> It's fine though, I found that s6 may start growing an IPC mechanism
>>> so we could possibly do a launchd/jobd style service later (ie to run
>>> things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
>>> arbitrary messages, etc) so I may try incorporating it again. :)
>>>
>>
>> Can't this be done with devd?
>
> Sure, but I'm looking for something more generic than just devd. Like,
> notifications about events like "default route is up" can be done by
> sniffing the rtsock, but notifications like "ntpdate has updated the
> date, we can now do crypto services" doesn't happen there right now.

devd was never intended to be limited to just device events from the
kernel. It has grown beyond that, and could easily grow to cope with
routing events and other notifications. No need to reinvent everything
for that.

What devd doesn't do is launchd / job control. That would be a whole
new kettle of fish for it, and one for which it may be ill suited. Though
viewed in the right way, it's all just a series of notifications: This service
is up, this is down, user wants to start this other one, etc, so maybe
it wouldn't so bad...

Warner
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Adrian Chadd]

On 25 August 2016 at 14:13, Warner Losh <imp@bsdimp.com> wrote:
> On Thu, Aug 25, 2016 at 3:08 PM, Adrian Chadd <adrian@freebsd.org> wrote:
>> On 25 August 2016 at 12:48, Lars Engels <lars.engels@0x20.net> wrote:
>>> On Thu, Aug 25, 2016 at 08:46:10AM -0700, Adrian Chadd wrote:
>>>> On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
>>>> <J.deBoynePollard-newsgroups@ntlworld.com> wrote:
>>>> > http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
>>>> > , Adrian Chadd:
>>>> >
>>>> >> We're using s6 at work, and it works out mostly ok. Mostly once you get
>>>> >> around the linuxisms, and the lack of sensible time code in it (its
>>>> >> calculations for daemon run duration is based on system time, not wall
>>>> >> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
>>>> >> hilarious), and some of the arcane bits to get logging working right.
>>>> >>
>>>> > What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
>>>> > Linuxisms, obviously.  But what Linuxisms does s6 have?
>>>>
>>>> We just had a bunch of fun trying to get it to build right, and the
>>>> uptime stuff really threw us.
>>>>
>>>> It's fine though, I found that s6 may start growing an IPC mechanism
>>>> so we could possibly do a launchd/jobd style service later (ie to run
>>>> things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
>>>> arbitrary messages, etc) so I may try incorporating it again. :)
>>>>
>>>
>>> Can't this be done with devd?
>>
>> Sure, but I'm looking for something more generic than just devd. Like,
>> notifications about events like "default route is up" can be done by
>> sniffing the rtsock, but notifications like "ntpdate has updated the
>> date, we can now do crypto services" doesn't happen there right now.
>
> devd was never intended to be limited to just device events from the
> kernel. It has grown beyond that, and could easily grow to cope with
> routing events and other notifications. No need to reinvent everything
> for that.

Right. I don't want to reinvent the wheel if it can be avoided.

But there are other things that want to produce and consume events.
eg, openvpn bringing up a VPN triggering possible ipfw rule changes.
Or openvpn coming down triggering other ipfw rule changes.

> What devd doesn't do is launchd / job control. That would be a whole
> new kettle of fish for it, and one for which it may be ill suited. Though
> viewed in the right way, it's all just a series of notifications: This service
> is up, this is down, user wants to start this other one, etc, so maybe
> it wouldn't so bad...

Well, ideally the jobd would sit on the message bus and take commands
to do things. Like dbus and udevd did in linux, before systemd
steamrolled over them. But then if I suggest we need a message bus
daemon up and going so arbitrary system pieces could talk to other
system pieces, I'll likely be shouted at.

But not by jkh. He'd likely be "YOURE ONLY JUST GETTING AROUND TO THIS
NOW?" and laugh a lot.

(jkh - please come to the next bafug so we can talk shop..)


-a
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Jan Bramkamp]

On 25/08/16 23:17, Adrian Chadd wrote:
> On 25 August 2016 at 14:13, Warner Losh <imp@bsdimp.com> wrote:
>> On Thu, Aug 25, 2016 at 3:08 PM, Adrian Chadd <adrian@freebsd.org> wrote:
>>> On 25 August 2016 at 12:48, Lars Engels <lars.engels@0x20.net> wrote:
>>>> On Thu, Aug 25, 2016 at 08:46:10AM -0700, Adrian Chadd wrote:
>>>>> On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
>>>>> <J.deBoynePollard-newsgroups@ntlworld.com> wrote:
>>>>>> http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
>>>>>> , Adrian Chadd:
>>>>>>
>>>>>>> We're using s6 at work, and it works out mostly ok. Mostly once you get
>>>>>>> around the linuxisms, and the lack of sensible time code in it (its
>>>>>>> calculations for daemon run duration is based on system time, not wall
>>>>>>> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
>>>>>>> hilarious), and some of the arcane bits to get logging working right.
>>>>>>>
>>>>>> What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init have
>>>>>> Linuxisms, obviously.  But what Linuxisms does s6 have?
>>>>>
>>>>> We just had a bunch of fun trying to get it to build right, and the
>>>>> uptime stuff really threw us.
>>>>>
>>>>> It's fine though, I found that s6 may start growing an IPC mechanism
>>>>> so we could possibly do a launchd/jobd style service later (ie to run
>>>>> things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
>>>>> arbitrary messages, etc) so I may try incorporating it again. :)
>>>>>
>>>>
>>>> Can't this be done with devd?
>>>
>>> Sure, but I'm looking for something more generic than just devd. Like,
>>> notifications about events like "default route is up" can be done by
>>> sniffing the rtsock, but notifications like "ntpdate has updated the
>>> date, we can now do crypto services" doesn't happen there right now.
>>
>> devd was never intended to be limited to just device events from the
>> kernel. It has grown beyond that, and could easily grow to cope with
>> routing events and other notifications. No need to reinvent everything
>> for that.
>
> Right. I don't want to reinvent the wheel if it can be avoided.

Afaik devd is limited handling events reported by the kernel on 
/etc/devctl. There is no way to inject events into arbitrary events from 
userspace into devd (no ptrace hacks don't count).

> But there are other things that want to produce and consume events.
> eg, openvpn bringing up a VPN triggering possible ipfw rule changes.
> Or openvpn coming down triggering other ipfw rule changes.

FreeBSD offers several IPC APIs but non of them can implemented reliable 
multicast as this would require an unbounded journal in stable memory. 
For most use-cases reliable notification of the current state is enough. 
Instead of reliably multicasting each message to each recipient just 
send each observer the latest state of each observed value e.g. in your 
OpenVPN example the IPFW wrapper doesn't care how many time the tunnel 
flapped. The user just wants the right firewall configuration for his 
current network environment. He doesn't want to replay every change on 
the way.

In macOS has notifyd offers this service. The optimizations in notifyd 
make it too large to just import its services into an init process, but 
having a simpler reliable notification mechanism early would be useful.

>> What devd doesn't do is launchd / job control. That would be a whole
>> new kettle of fish for it, and one for which it may be ill suited. Though
>> viewed in the right way, it's all just a series of notifications: This service
>> is up, this is down, user wants to start this other one, etc, so maybe
>> it wouldn't so bad...
>
> Well, ideally the jobd would sit on the message bus and take commands
> to do things. Like dbus and udevd did in linux, before systemd
> steamrolled over them. But then if I suggest we need a message bus
> daemon up and going so arbitrary system pieces could talk to other
> system pieces, I'll likely be shouted at.
>
> But not by jkh. He'd likely be "YOURE ONLY JUST GETTING AROUND TO THIS
> NOW?" and laugh a lot.
>
> (jkh - please come to the next bafug so we can talk shop..)

Process spawning and supervision should be separate from the policy 
engine(s) as the process supervision graph should be a tree rooted in 
pid 1, but the user might want to run multiple rule/policy engines. An 
init process should just offer the required mechanisms and nothing more. 
Convenient policies can be implemented on top of those mechanisms.

For my own init system I'm still not sure if the init process should 
track services and their dependencies at all or just processes and keep 
the concept of services and dependencies in a service management process 
instead of the init process.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Warner Losh]

On Fri, Aug 26, 2016 at 4:44 AM, Jan Bramkamp <crest@rlwinm.de> wrote:
> On 25/08/16 23:17, Adrian Chadd wrote:
>>
>> On 25 August 2016 at 14:13, Warner Losh <imp@bsdimp.com> wrote:
>>>
>>> On Thu, Aug 25, 2016 at 3:08 PM, Adrian Chadd <adrian@freebsd.org> wrote:
>>>>
>>>> On 25 August 2016 at 12:48, Lars Engels <lars.engels@0x20.net> wrote:
>>>>>
>>>>> On Thu, Aug 25, 2016 at 08:46:10AM -0700, Adrian Chadd wrote:
>>>>>>
>>>>>> On 24 August 2016 at 21:53, Jonathan de Boyne Pollard
>>>>>> <J.deBoynePollard-newsgroups@ntlworld.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> http://adrianchadd.blogspot.co.uk/2016/08/freebsd-on-tiny-system-whats-missing.html?showComment=1471236502051#c1305086913155850955
>>>>>>> , Adrian Chadd:
>>>>>>>
>>>>>>>> We're using s6 at work, and it works out mostly ok. Mostly once you
>>>>>>>> get
>>>>>>>> around the linuxisms, and the lack of sensible time code in it (its
>>>>>>>> calculations for daemon run duration is based on system time, not
>>>>>>>> wall
>>>>>>>> clock, so if your box boots jan 1, 1970 then gets NTP, things are..
>>>>>>>> hilarious), and some of the arcane bits to get logging working
>>>>>>>> right.
>>>>>>>>
>>>>>>> What are these Linuxisms in s6?  s6-linux-utils and s6-linux-init
>>>>>>> have
>>>>>>> Linuxisms, obviously.  But what Linuxisms does s6 have?
>>>>>>
>>>>>>
>>>>>> We just had a bunch of fun trying to get it to build right, and the
>>>>>> uptime stuff really threw us.
>>>>>>
>>>>>> It's fine though, I found that s6 may start growing an IPC mechanism
>>>>>> so we could possibly do a launchd/jobd style service later (ie to run
>>>>>> things upon event changes, like ifup, ifdown, ifcreate, ifdestroy,
>>>>>> arbitrary messages, etc) so I may try incorporating it again. :)
>>>>>>
>>>>>
>>>>> Can't this be done with devd?
>>>>
>>>>
>>>> Sure, but I'm looking for something more generic than just devd. Like,
>>>> notifications about events like "default route is up" can be done by
>>>> sniffing the rtsock, but notifications like "ntpdate has updated the
>>>> date, we can now do crypto services" doesn't happen there right now.
>>>
>>>
>>> devd was never intended to be limited to just device events from the
>>> kernel. It has grown beyond that, and could easily grow to cope with
>>> routing events and other notifications. No need to reinvent everything
>>> for that.
>>
>>
>> Right. I don't want to reinvent the wheel if it can be avoided.
>
>
> Afaik devd is limited handling events reported by the kernel on /etc/devctl.
> There is no way to inject events into arbitrary events from userspace into
> devd (no ptrace hacks don't count).

It would be trivial to add though, especially since the author is easy to
work with :). It already fans out data to multiple listeners.

Warner
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Jonathan de Boyne Pollard]

Jonathan de Boyne Pollard:

> What are these Linuxisms in s6? s6-linux-utils and s6-linux-init have 
> Linuxisms, obviously. But what Linuxisms does s6 have?
>
Adrian Chadd:

> We just had a bunch of fun trying to get it to build right, [...]
>
Such as what, specifically?

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Jonathan de Boyne Pollard]

Adrian Chadd:

> Sure, but I'm looking for something more generic than just devd. Like, 
> notifications about events like "default route is up" can be done by 
> sniffing the rtsock, but notifications like "ntpdate has updated the 
> date, we can now do crypto services" doesn't happen there right now.
>
You're reinventing upstart.  The lesson of upstart is that whilst the 
event-driven paradigm looks like the bright shiny future, once one gets 
down to the details it is a lot harder than it at first appears.  I 
strongly recommended learning about upstart, and especially learning the 
problems that people hit with it, to anyone going down the same route.  
The Debian systemd Hoo-Hah had some lengthy discussion of upstart.

(I regret not having bookmarked the discussion that I once came across, 
where someone opined that xe preferred systemd to upstart because at a 
Linux conference the systemd presentation had been exciting and had been 
put forward as the wave of the future, where upstart had been presented 
as old-school, traditional, and boring.  Ironically, this person wasn't 
aware that the designs are exactly the opposite of that.  upstart has 
the novel event-driven design where the system is configured with the 
information that event A triggers programs P, Q, and R, and the system 
starts by raising a "first event", that runs programs, that raise 
further events, that run further programs.  Whereas it is systemd that 
has the conventional design, shared by Mewburn rc and others, of 
starting from a goal, working through a dependency tree, and doing 
topological sorts.)

The Debian people chose to improve a non-event-driven architecture 
instead.  It's a lesson to be learned from SMF, in fact.  One can have a 
lot more additional abstract targets, such as "/milestone/name-services" 
and "/milestone/system-clock", and dependencies to and from them.  The 
world is not 2 to 4 run levels plus "DAEMON", "NETWORKING", and "$local-fs".

That said, something like this hypothetical "/milestone/system-clock" is 
a milestone that would need to be reached *very* early on in the 
bootstrap process.  Fixing up the clock is something that both the nosh 
system manager and systemd handle themselves directly, outwith of 
service management.  More on this in a moment.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Jonathan de Boyne Pollard]

Adrian Chadd:

> [...] the uptime stuff really threw us.
>
It's unfair to lay such system time problems at s6's door. Systems whose 
system clock jumps 46 years during system bootstrap don't get to blame 
s6 for mad time gaps that appear in logs and service start time 
records.  There is a *lot* of the Unix and Linux worlds that depends 
from time being right.  It's not just s6 that is affected by such 
things.  You note crypto.  There are a lot of other things as well that 
have unstated, sometimes undocumented, and sometimes surprising 
dependencies upon system time being current.

Here's one such.

For quite a while, Linux distributions had rather an odd problem at 
bootstrap.  They'd repeatedly fsck volumes at every bootstrap when they 
need not have.  And this didn't affect U.S. or U.K. people, which is in 
part why it persisted for so long.

* https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/63175

* https://bugs.archlinux.org/task/17438

* http://lwn.net/Articles/264498/

The problem was that people were erroneously running their real-time 
clocks in local time rather than UTC, and this triggered an odd hidden 
dependency upon having the right time in the system clock in countries 
where local time was in advance of UTC.  The Linux method for handling 
RTCs erroneously running in local time is for the system bootstrap to 
make a special settimeofday() call that effectively tells the kernel 
what the UTC offset is for the RTC hardware.  This could happen *after* 
the fsck of the root volume, however.  So whilst that fsck was 
happening, the kernel was assuming that UTC was the local time that it 
had taken from the RTC and initialized its system clock with.  In 
effect, as soon as the special settimeofday() call was executed, the 
system clock would jump backwards by one or more hours, to what UTC 
actually was.

But the ext2/3/4 filesystem format has last checked/mounted/written 
timestamps in its superblock.  Part of the checking to see whether a 
full fsck is needed at bootstrap is comparing them to the current time.  
If they are in the future by hours or more, something is clearly wrong, 
thinks fsck, and it runs the full check.  At bootstrap, when the initial 
fsck (of at least the root volume and sometimes other volumes as well) 
is run, the system clock is not UTC yet.  Comedy results.

Both systemd and the nosh system manager have to ensure that they do the 
special settimeofday() system call before they start off service 
management and thus run mount/fsck services, or indeed anything else 
that might have a closet dependency from not stepping the system time by 
hours partway through bootstrap.  The nosh system-manager's manual page 
has a whole section on this subject.

FreeBSD/PC-BSD has a mechanism for correctly reading a RTC that is 
erroneously in local time.  One sets up the RTC's offset from UTC in the 
machdep.adjkerntz variable in /boot/loader.conf{,.local} and the system 
clock never has to jump by hours during bootstrap.  I've yet to 
experience a FreeBSD/PC-BSD system where the installer actually 
configures this, though.

Interestingly, FreeBSD/PC-BSD also has a fallback mechanism that uses 
the latest volume mount timestamp that it can find as the initial system 
time when no hardware clock device registers at bootstrap.  Presumably 
you have a clock device that registers but it is not battery-backed, 
your volumes don't preserve (or reset) their mount timestamps, or you 
are encountering the comedy situation where FreeBSD/PC-BSD is setting 
the system clock to 1970-01-01 because the last time around it mounted 
the filesystems before the clock was corrected.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: Linuxisms in s6

[Adrian Chadd]

On 27 August 2016 at 09:37, Jonathan de Boyne Pollard
<J.deBoynePollard-newsgroups@ntlworld.com> wrote:
> Adrian Chadd:
>
>> Sure, but I'm looking for something more generic than just devd. Like,
>> notifications about events like "default route is up" can be done by
>> sniffing the rtsock, but notifications like "ntpdate has updated the date,
>> we can now do crypto services" doesn't happen there right now.
>>
> You're reinventing upstart.  The lesson of upstart is that whilst the
> event-driven paradigm looks like the bright shiny future, once one gets down
> to the details it is a lot harder than it at first appears.  I strongly
> recommended learning about upstart, and especially learning the problems
> that people hit with it, to anyone going down the same route.  The Debian
> systemd Hoo-Hah had some lengthy discussion of upstart.

Oh yeah, I'm aware of the differences between systemd and upstart.

> (I regret not having bookmarked the discussion that I once came across,
> where someone opined that xe preferred systemd to upstart because at a Linux
> conference the systemd presentation had been exciting and had been put
> forward as the wave of the future, where upstart had been presented as
> old-school, traditional, and boring.  Ironically, this person wasn't aware
> that the designs are exactly the opposite of that.  upstart has the novel
> event-driven design where the system is configured with the information that
> event A triggers programs P, Q, and R, and the system starts by raising a
> "first event", that runs programs, that raise further events, that run
> further programs.  Whereas it is systemd that has the conventional design,
> shared by Mewburn rc and others, of starting from a goal, working through a
> dependency tree, and doing topological sorts.)

For some environments (servers, desktop environments, etc) where most
of the dynamicness comes from "which user is logged in" and "maybe I
don't have my network hardware plugged in until later", sure, I can
see that the dependency tree model works great. Your aim is some
grander set of checkpoints, like "What do I need to run basic network
services", "What do i need up to run a desktop login environment",
etc.

But for things like "I'm a vpn server, and I need to speak to a vpn
server to provide access to my vpn clients, oh and I have firewall
rules that need applying based on which dynamic interfaces come/go" it
still maps to an event driven mechanism. Sure you can map that event
driven mechanism to a set of targets, but those targets may be
per-interface. Like, when a vpn client interface comes up, I have a
set of things that need to happen that depend upon which client. Same
with wireless clients associating. I may hit some situations (eg above
a certain threshold of associated clients) where I trigger events such
as "clean up old clients", "look at migrating clients to other access
points", etc.

If I can do this with s6, then cool - please let me know how and I'll
re-consider it. But regardless of that, I also do need some
generalised dbus style mechanism so all the pieces of the system can
talk to the other pieces of the system without having to .. well, wrap
it all in 'service' style shell scripts and calling ifconfig wlanX
list sta periodically from everywhere.

> The Debian people chose to improve a non-event-driven architecture instead.
> It's a lesson to be learned from SMF, in fact.  One can have a lot more
> additional abstract targets, such as "/milestone/name-services" and
> "/milestone/system-clock", and dependencies to and from them.  The world is
> not 2 to 4 run levels plus "DAEMON", "NETWORKING", and "$local-fs".
>
> That said, something like this hypothetical "/milestone/system-clock" is a
> milestone that would need to be reached *very* early on in the bootstrap
> process.  Fixing up the clock is something that both the nosh system manager
> and systemd handle themselves directly, outwith of service management.  More
> on this in a moment.

So, this is where it gets exciting in some of these appliances.

Sometimes there's no 'date/time' RTC hardware. Sometimes, you have to
present some UI to the user so they can enter a date/time, and some
services need to run before that, but some services (notably ntpdate)
won't work. So we can't, say, hold all network services back until we
have valid date/time or a bunch of the UI infrastructure won't be
there.

I can't expect the whole system to stop waiting for a system-clock
time to be valid. In fact, i ended up adding some stuff to our
appliance images that store the current clock value in a file every 15
minutes - the rootfs is read-only, so i can't just boot up from /its/
concept of "last mounted", as that filesystem is not modifiable.
Trouble is, that gets read from the system during boot, after FSes
have been mounted, etc. It's all terrible.

Thanks,



-adrian
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"