nosh version 1.14

nosh version 1.14

[Jonathan de Boyne Pollard]

nosh is now up to version 1.14

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

These particular changelog entries are a big deal for Debian Linux.

   * The previous regular sockets, sysinit services, and standard 
targets packages are now all merged into the bundles package.
   * There is now a new user VT package that runs new-style user VT 
services.
   * There is now a new kernel VT package that runs old-style kernel VT 
services.
   * The nosh-systemd-services package now automatically runs various 
nosh subsystems under systemd.

I've taken the list of Debian packages available for download off the 
infobox at the side of the blurb page and set up a page devoted 
specifically to the Debian packages, explaining what they contain and 
are for in more detail than could fit into that infobox.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/debian-binary-packages.html

There's a known bug that I'm working on.  Because the preset conversion 
is a little too aggressive, and because getttynam() exists on Debian 
Linux, you'll currently need a dummy BSD /etc/ttys file that presets the 
ttylogin@.service services that you want. Something like:

     tty1 /bin/false linux on secure
     tty5 /bin/false linux on secure
     tty7 /bin/false linux off secure
     vc0-tty /bin/false linux on secure
     vc1-tty /bin/false linux on secure
     vc2-tty /bin/false linux on secure

Those "vcN-tty" lines are user-mode virtual terminals.  As you can 
probably work out from this, on the BSD side the nosh user-mode virtual 
terminal system is able to pull terminal login service enable/disable 
information from the existing /etc/ttys configuration mechanism.  (It 
also can pull from /etc/rc.conf and does "onifconsole" too.)

The list of 157 things that I have to convert in order to fully replace 
BSD /etc/rc.d is discussed in detail on the FreeBSD Hackers mailing 
list.  We have long since passed the point where it's possible to have 
an entirely nosh-managed FreeBSD/PC-BSD system, though.  The list of 
things that I have to convert before I can likewise run my Debian Linux 
system fully under the nosh system-manager is down to about a handful, 
and is mainly the likes of service bundles for dbus and udev.  If you 
already have daemontools-style run scripts for those, or eudev, or mdev, 
or whatever you enjoy, then you can race ahead of me.  (-:

The page also warns about the nosh-bundles package potentially enabling 
a lot of services.  It's a package of over 400 service bundles.  One way 
of avoiding this is to go the only-enable-what-I-permit route, and use 
this 99-default.preset:

     disable *.service
     disable *.socket

With something like this 00-administrator.preset alongside:

     enable    cyclog@acpid.service
     enable    cyclog@atd.service
     enable    cyclog@console-fb-realizer@*.service
     enable    cyclog@console-multiplexor@*.service
     enable    cyclog@gnucron.service
     enable    cyclog@kerneloops.service
     enable    cyclog@ModemManager.service
     enable    cyclog@NetworkManager.service
     enable    cyclog@org.cups.cups*.service
     enable    cyclog@polkitd.service
     enable    cyclog@terminal-emulator@*.service
     enable    cyclog@ttylogin@*.service
     enable    cyclog@update-binfmts.service
     enable    cyclog@wpa_supplicant.service
     enable    acpid.service
     enable    atd.service
     enable    console-fb-realizer@*.service
     enable    console-multiplexor@*.service
     enable    gnucron.service
     enable    kerneloops.service
     enable    ModemManager.service
     enable    NetworkManager.service
     enable    org.cups.cups*.service
     enable    polkitd.service
     enable    terminal-emulator@*.service
     enable    ttylogin@*.service
     enable    update-binfmts.service
     enable    wpa_supplicant.service

Adjust according to taste, of course.  Mine also enables various 
additional service bundles including dnscache, tinydns, http6d, 
rabbitmq-server, and epmd (and their concomitant logging services) for 
example.

nosh version 1.16

[Jonathan de Boyne Pollard]

nosh is now up to version 1.16

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

As you'll see, the WWW pages have expanded a bit.  In part this is 
because of the Big News, which is the arrival of FreeBSD packages, 
bringing FreeBSD up to par with Debian.  The old box down the right-hand 
side of the page was starting to make the thing look lop-sided.  (-:

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/freebsd-binary-packages.html
* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/debian-binary-packages.html

More big news on the package front is the reorganization into a main 
"bundles" package and a group of "-run" packages.  Lesser news is the 
addition of packages for enabling/running various further groups of 
services.

Comparatively small news are things like the change to the output of 
"system-control status" and "service-status", which now uses long form 
that displays more information.  "svstat" retains its 1 line short form, 
however.  There's also a "system-control cat" command, for dumping out 
service bundle configuration files.  The new "emergency-login" fills the 
gap where FreeBSD lacks a "sulogin" (because it's hardwired into the old 
"init" and thus unusable separately), and also means that there's no 
need to rely upon the old System 5 utilities/Linux utilities for 
"sulogin" on Linux.

There is also a new roadmap WWW page.  The Nosh Guide has also gained 
several new pages dealing with logging and the import of external stuff.


_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

redo version 1.2

[Jonathan de Boyne Pollard]

redo is now at version 1.2

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/redo.html

Two changes.  First, as you can see, there's now a pre-built FreeBSD 
binary package.

The second change is something that has been annoying me for some 
while.  Sometimes, especially when one is building packages, one ends up 
with redo invoked by make. GNU make puts only flags in the MAKEFLAGS 
environment variable.  BSD make, however, puts at least one macro 
definition in there too.  redo was complaining about that, because it 
was only expecting flags.  Now it knows to ignore macro definitions.  (-:

You'll need redo for one of the new (optional, of course) mechanisms in 
nosh version 1.17.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

redo version 1.4

[Jonathan de Boyne Pollard via freebsd-hackers]

redo is now at version 1.4

  * http://jdebp.eu./Softwares/redo/

The only change from 1.3 is a belt-and-braces protection mechanism that 
prevents cleanup code from being told to delete a parent directory.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: nosh version 1.17

[Jonathan de Boyne Pollard]

nosh is now up to version 1.17

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

As I said to gdiazhartusch a while back, and as listed on the roadmap 
page, there is now a tool for auto-creating the mount@*, fsck@*, 
ttylogin@*, dump@*, and swap@* service bundles from /etc/ttys and 
/etc/fstab, and propagating settings from /etc/rc.conf{,.local} to 
service bundle env/ directories.  This is documented in the Nosh Guide, 
in the section on external formats. It involves the use of redo, as I 
just mentioned in the release annoucement for redo 1.2, because it's a 
suite of redo scripts that endeavour to re-import the settings only when 
the source files change.  An important design choice, called out in the 
introduction to the Nosh Guide, is that this is not a system of 
continually-regenerated ephemera.

The Nosh Guide introduction has been completely rewritten, by the way.

The external formats import relies upon rcctl, which now has the full 
implementation of "set" and "get" that was promised in the roadmap.  
Note the caveat in the manual about not using "set" on a "status" 
variable.  That's an OpenBSD idiosyncrasy that wouldn't be correct for 
FreeBSD even with the original tool.  Use "enable" and "disable".  
They'll work.

Part of importing /etc/fstab is generating appropriate dump@* and swap@* 
service bundles.  You'll note that swapauto and swaplate are no longer 
services but targets.  As services, they relied upon options to "swapon" 
and were one giant lump.  As targets, they pull in all of the generated 
swap@* services which are individually enabled.  Also note that whilst 
the pre-built mount@-, fsck@-, mount@-usr, fsck@-usr, mount@-var, and 
fsck@-var service bundles are still supplied, the auto-creation system 
(or something similar) is meant to supersede them with bundles that 
match your /etc/fstab after system installation.

Those aren't the only service bundle changes.  There's the steady 
chipping away at that list of 157 rc.d scripts, which has just lost off 
its list a few pf services and savecore amongst other things. You might 
be wondering why mountcritlocal and mountcritremote are still there.  
The new auto-creation system makes mount@* and fsck@* bundles, and links 
them into targets, after all.  The answer is that those FreeBSD rc.d 
scripts (like several others) have had some feature creep, and mounting 
is not all that they in fact do. There's some temporary files cleanup in 
there, for example.

Which leaves just the last major change, which is the /etc/ 
subdirectories.  /etc/system-manager/targets and /etc/sv have been 
brought together as /etc/service-bundles/targets and 
/etc/service-bundles/services.  And /etc/system-manager/presets is now 
the better named /etc/system-control/presets as this is a function of 
the latter tool not of the former.  The post-upgrade scripts in the 
binary packages will move the directories and create appropriate 
symbolic links in the right places.  If you are building from source and 
hand-installing ... well look at what the post-upgrade scripts do, since 
they are right there in the source package, and do that.  (-:
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

pflog(8) manual page bug

[Jonathan de Boyne Pollard]

Jonathan de Boyne Pollard:
> There's the steady chipping away at that list of 157 rc.d scripts, 
> which has just lost off its list a few pf services and savecore 
> amongst other things.

Yet another bug report that has come out of this is that the pflogd(8) 
manual page has some gibberish under the -i option.

The original OpenBSD update ...

* 
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/pflogd/pflogd.8.diff?r1=1.35&r2=1.36&f=h

... was not accurately brought over to FreeBSD ...

* 
https://svnweb.freebsd.org/base/head/contrib/pf/pflogd/pflogd.8?r1=171172&r2=223637

... and was missing two lines that make the gibberish make sense.

Of course ...

* 
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/pflogd/pflogd.8.diff?r1=1.37&r2=1.38&f=h

... that option doesn't even exist any more.  (-:
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: pflog(8) manual page bug

[Warren Block]

On Sun, 28 Jun 2015, Jonathan de Boyne Pollard wrote:

> Jonathan de Boyne Pollard:
>> There's the steady chipping away at that list of 157 rc.d scripts, which 
>> has just lost off its list a few pf services and savecore amongst other 
>> things.
>
> Yet another bug report that has come out of this is that the pflogd(8) manual 
> page has some gibberish under the -i option.
>
> The original OpenBSD update ...
>
> * 
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/pflogd/pflogd.8.diff?r1=1.35&r2=1.36&f=h
>
> ... was not accurately brought over to FreeBSD ...
>
> * 
> https://svnweb.freebsd.org/base/head/contrib/pf/pflogd/pflogd.8?r1=171172&r2=223637
>
> ... and was missing two lines that make the gibberish make sense.
>
> Of course ...
>
> * 
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sbin/pflogd/pflogd.8.diff?r1=1.37&r2=1.38&f=h
>
> ... that option doesn't even exist any more.  (-:

Fixed in r284914.  Thanks!
_______________________________________________
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.18

[Jonathan de Boyne Pollard]

nosh is now up to version 1.18

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

The big news for this release is the nosh-run-system-manager Debian 
binary package.  This, and the new additional service bundles in 
nosh-bundles, package up everything that is needed for running an 
entirely nosh-managed basic Debian system with the nosh system-manager 
program as process #1.  And so the entry on the roadmap WWW page is 
crossed out.  Some notes:

* Don't forget that the Nosh Guide has a whole chapter on troubleshooting.
* With that package alone, you get very little running.  This is 
intentional.  You'll have to install other nosh-run packages, or add 
presets, for the various other things that you want.  To get an OpenSSH 
server running, for example, you'll need a local preset file (named, 
say, /etc/system-control/presets/20-sshd.preset) with "enable sshd" and 
"enable cyclog@sshd" before (re-)installing nosh-bundles.  
(Re-)Installing the nosh-bundles package (re-)applies all current 
presets, including your local ones, and auto-starts all enabled services.
* If you are running the freedesktop services, read the notes 
hyperlinked-from the package download page.
* You may have spotted that there's a choice between running udev and 
busybox mdev.  (You pretty much must run one or the other for a fully 
functional system.)  The nosh-run-busybox-mdev package is broken.  I 
forgot to write the adapter tool.  I've written it ready for version 
1.19.  There will be more said on the subject of busybox mdev in the 
1.19 announcement, therefore.
* It's also intentional that you don't get System 5 shim commands for 
the likes of "telinit" and "halt" unless you install the 
nosh-systemv-shims package.  "system-control poweroff" works without the 
presence of the shims, of course.
* For novices, I recommend starting with nosh-run-kernel-vt . 
nosh-run-user-vt still requires a manual step, after re-building the 
service configuration each time, of "system-control disable 
ttylogin@tty{1,2,3,4,5,6,7,8,9,10,11,12}".
* The recovery mode misbehaviour is a known problem.  I'm 
investigating.  As a local fix, boot with init=/bin/sh on the kernel 
command line and then run "exec /sbin/init -s" or even "exec /sbin/init 
-b" from that shell prompt.

This is not the only news, of course.  The BSD crowd should not feel 
left out, moreover.

There are four long-standing problems with the Linux libkqueue library.  
One of those problems causes svscan a.k.a. service-dt-scanner to be 
spuriously woken up.  This doesn't affect Debian but does affect Linux 
operating systems such as Gentoo that have more recent versions of that 
library.  This has been worked around in version 1.18.

The pre-built mount@-, fsck@-, mount@-usr, fsck@-usr, mount@-var, and 
fsck@-var service bundles have been removed.  Generation of the service 
bundles for mounting and checking volumes is now entirely based upon the 
auto-creation system in /etc/system-control/convert/ .  If you are 
installing from scratch by hand, then you must remember to "redo all" in 
that directory.  The nosh-bundles package does this for you as part of 
its post-install procedures.

The problem with the local-syslog-read service on Linux providing the 
wrong socket (the BSD one) has been fixed.

The tools now speak true TAI, rather than UTC-10.  There's an 
explanation of the consequences of this in the manual pages for cyclog, 
tai64n, and tai64nlocal.

The /etc/fstab conversion system now recognizes remote filesystem types 
and attaches the generated services to remote-fs.target .
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.19

[Jonathan de Boyne Pollard]

nosh is now up to version 1.19

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

The important news is that the embarrassment with the post-install setup 
script for the Linux nosh-run-kernel-vt package is fixed.  It was a 
missing 1-line escape() shell function.  I apologize.

Other terminal management news is that there's now a console-clear 
command that does pretty much the same thing as the Bourne Again shell's 
clear_console command (also coming with that name as a symbolic link 
alias) but better.

* The bash clear_console tries to open a lot of device files, as can be 
seen in Ubuntu bug #39068.  This tool by comparison doesn't need 
anything more than its standard output, and doesn't attempt to open any 
terminal devices itself at all.
* The bash tool is specific to the Linux kernel terminal emulator. It 
had to be turned off for Debian kFreeBSD in Debian bug #355336, patched 
to make it stop when run as the superuser in xterm in Debian bug 
#355815, and worked around again in Debian bug #793883.  This tool, 
contrastingly, actually works with xterm and PuTTY and clears their own 
scrollback buffers.  It uses a different mechanism that both they and 
(ironically) the Linux kernel terminal emulator since 2011, all support.
* Debian bug #791342 would be fixed by it, because it doesn't use the 
bodge of attempting to switch virtual terminals away and back (using 
either tty1 or tty2 as the "other" terminal) to clear the scrollback buffer.

On the gripping hand, this is something that one doesn't actually need 
if one is using the nosh-run-user-vt package. console-terminal-emulator 
supports the same extension to ECMA-48 Erase Display as xterm and PuTTY 
do, but the raison d'être for clear_console is the likes of Debian bug 
#331504.  clear_console is in fact a ten-year-old bodge, addressing a 
security/privacy concern that's a lot older still.  With user-space 
virtual terminals, one has the freedom to do things right, without the 
need for such bodges.  (-:  As the console-terminal-emulator manual page 
explains, when a login session terminates and the terminal is hung up, 
the terminal emulator erases the whole display buffer.

In more other news: On Linux, fsck at bootstrap time is now monitored.  
What this means from a user standpoint is that if your system reaches 
its "maximum mounts before a forced full fsck" point, it doesn't just 
sit there with nothing visibly happening (if one cannot see the hard 
disc activity light) for ages.  The fsck@* services now invoke 
"monitored-fsck" rather than fsck directly. This is an ordinary 
chain-loading tool that opens a client connection to a local (i.e. 
AF_LOCAL) socket and then chains to fsck adding in its (Linux-specific) 
-Cfd option.   There's a new monitor-fsck-progress service that runs the 
server for that socket, and displays progress information on the 
console.  This latter is intentionally replaceable by alternative 
services, of course.  I'm intending to make its output somewhat 
prettier, rather than just dumping the raw information at you as it does 
in this release.  But if you want to write your own ...

You'll have to delete /etc/system-control/convert/volumes (or modify the 
contents of /etc/fstab) and run "redo all" to get your existing 
auto-created fsck@* service bundles regenerated with the new command.  
Or just edit the run files replacing fsck with monitored-fsck .

The big news is that as promised in the 1.18 announcement the 
nosh-run-busybox-mdev package is now functional.  Also as promised in 
that announcement, here's more on the subject.

The nosh toolset doesn't come with a bunch of rules for your 
plug-and-play manager, be that (BSD) devd, (Linux) udev, or busybox 
mdev.  Your plug-and-play manager does, or should do.  As packaged up 
for Debian Linux, udev comes with a whole bunch of pre-supplied rules in 
/lib/udev/rules/ that gets one the "usual" device file tree in /dev/ .  
And it almost goes without saying that the BSDs come with devd rules in 
the box.  The same is not true for the busybox Debian package.  There's 
no /etc/mdev.conf supplied.

You MUST write one before using busybox mdev.  busybox mdev's default 
behaviour as packaged, in the absence of /etc/mdev.conf , may be logical 
and straightforward; but it does not result in a working Debian system.  
Some things that I've hit myself are /dev/null not being accessible to 
anyone except the superuser, which affects loads of things all over the 
shop, and event device files not being under /dev/input/ where other 
parts of the system expect them to be.

There's plenty to read on this subject in the non-Debian world, starting 
with but not limited to:

* https://wiki.gentoo.org/wiki/Mdev
* http://linuxfromscratch.org/clfs/view/clfs-3.0/mips/bootscripts/mdev.html

You'll have to adapt these for Debian.  There are also the examples in 
/usr/share/doc/busybox/examples/ , of course, the larger of the two 
fixing both of the aforementioned problems.  The positive news is that 
the busybox-mdev service implicitly serializes invocations of mdev, so 
that there's no need for mucking around with mdev's sequence number 
mechanism.

The recovery mode problem mentioned in the 1.18 announcement turns out 
to have a simple local fix, which I'll incorporate into a more 
comprehensive service fix:

# ln -s rescue /etc/service-bundles/targets/single
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.20

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.20 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

It's worth noting that the WWW site has gained some more pages, an 
installation how-to and a quick look at user-space virtual terminals.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html
* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/user-vt-screenshots.html

The command and tool list page, which was woefully out of date, has had 
some attention, too.  It is rather longer than it was.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html

You might notice a couple of new BSD packages, as well. FreeBSD/PC-BSD 
binary packaging is now up to parity with Debian Linux.  One can create 
a fully-nosh-managed system on both just by installing some binary packages.

This wipes another to-do item off the roadmap page.  The list of 
remaining rc.d items on the roadmap has shrunk, also.  As always, 
assistance in wiping those remaining rc.d items off the list is 
welcome.  If someone feels up to tackling /etc/rc.d/bluetooth, perhaps 
looking at what Iain Hibbert has apparently already done, for example ...

In addition to having yet more service bundles, this release irons out 
some wrinkles in startup and shutdown.  The sysinit phase of bootstrap 
was causing undesirable mounts in emergency mode.  That has been 
restructured.  Some ordering problems in shutdown relating to unmounting 
filesystems have also been fixed.  And the System 5/BSD compatibility 
reboot, halt, and poweroff shims no longer rely upon some other 
toolset's (not necessarily even present) shutdown command.

There are now -run packages for four different Debian Linux 
plug-and-play managers, with vdev and suckless mdev now added.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: nosh version 1.20

[Joe Maloney]

Hi,
do you have a source code repository somewhere for nosh?  Like on GitHub?

Joe Maloney


On Sun, Sep 27, 2015 at 8:05 PM, Jonathan de Boyne Pollard <
J.deBoynePollard-newsgroups@ntlworld.com> wrote:

> The nosh package is now up to version 1.20 .
>
> *
> http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
>
> It's worth noting that the WWW site has gained some more pages, an
> installation how-to and a quick look at user-space virtual terminals.
>
> *
> http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/timorous-admin-installation-how-to.html
> *
> http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/user-vt-screenshots.html
>
> The command and tool list page, which was woefully out of date, has had
> some attention, too.  It is rather longer than it was.
>
> *
> http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/commands.html
>
> You might notice a couple of new BSD packages, as well. FreeBSD/PC-BSD
> binary packaging is now up to parity with Debian Linux.  One can create a
> fully-nosh-managed system on both just by installing some binary packages.
>
> This wipes another to-do item off the roadmap page.  The list of remaining
> rc.d items on the roadmap has shrunk, also.  As always, assistance in
> wiping those remaining rc.d items off the list is welcome.  If someone
> feels up to tackling /etc/rc.d/bluetooth, perhaps looking at what Iain
> Hibbert has apparently already done, for example ...
>
> In addition to having yet more service bundles, this release irons out
> some wrinkles in startup and shutdown.  The sysinit phase of bootstrap was
> causing undesirable mounts in emergency mode.  That has been restructured.
> Some ordering problems in shutdown relating to unmounting filesystems have
> also been fixed.  And the System 5/BSD compatibility reboot, halt, and
> poweroff shims no longer rely upon some other toolset's (not necessarily
> even present) shutdown command.
>
> There are now -run packages for four different Debian Linux plug-and-play
> managers, with vdev and suckless mdev now added.
> _______________________________________________
> freebsd-hackers@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
>
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.20

[Jonathan de Boyne Pollard]

Joe Maloney:
> do you have a source code  repository somewhere for nosh?

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/source-package.html

The source package and how to build from source are here.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.21

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.21 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html

console-terminal-emulator now has a mouse input event protocol, and 
speaks both the DEC VT Locator protocol and the xterm Private Mode 1006 
protocol over the terminal interface to applications.  These are the 
protocols that you get with ttymouse=dec and ttymouse=sgr in vim.  Since 
support for the 1006 protocol is fairly widespread in the relevant 
places nowadays, it seemed not worthwhile implementing the inferior 
Private Mode 1005 and Private Mode 1015 protocols. Moreover, 
console-terminal-emulator is UTF-8 and the Private Mode 1005 protocol 
has ambiguities once one introduces UTF-8.

console-fb-realizer as yet only talks to real mouse input devices on 
Linux, but handling FreeBSD/PC-BSD mouse input devices is on the 
roadmap.  This has necessitated a change in the command-line syntax of 
console-fb-realizer, and concomitant changes in the pre-supplied 
realizer service bundle, which will need to be updated in tandem if you 
are using user-space virtual terminals.  This change allows the mouse 
input device to be specified in addition to the keyboard input event 
device.  It also slightly regularizes display-only mode, which is 
signified now by the simple lack of any mouse or keyboard device 
specifications, rather than by an explicit option.  Yes, I am aware that 
there's no mouse cursor sprite drawn yet.  Acutely so.

FreeBSD improvements include the completion of geli and gbde import that 
the sharp-eyed might have noticed quietly part-done in version 1.20.  
/etc/fstab entries for volumes using these should now be translated into 
appropriate interdependent mount@*, gbde@*, and geli@* service bundles.  
This is rather difficult for me to test, though, as noted on the roadmap.

The big PC-BSD improvement is jails support, which has lurked at the 
bottom of the roadmap page for a while.  Both PC-BSD Warden and FreeBSD 
9 jails are now recognized by the external configuration import 
subsystem, and converted into appropriate service bundles. The mechanism 
here is fairly straightforward:  The jails themselves are one service 
bundle, and the programs that run in the jails are another.  The latter 
service is after/ and wants/ the former service.  The jexec command is a 
chain loading tool that modifies process state in the same vein as 
setuidgid, softlimit, and envdir, and one can simply employ it as such.  
The jail command can be used analogously, with jail -c and jail -r, to 
how the mount and umount commands in mount@* service bundles are used.  
Those are what the import subsystem does.

Importing Warden Linux jails isn't available yet; and some of the more 
esoteric FreeBSD 9 rc.conf and PC-BSD Warden METADIR/* jail options are 
not yet imported.  Enabling jails to be autostarted at bootstrap is via 
the "jails" and "warden" targets, by the way.

The rc.d conversion project has progressed, with a few more things wiped 
off.  As mentioned in the version 1.20 message, all assistance in wiping 
the final 40-odd FreeBSD rc.d scripts off the list, to be found on the 
roadmap page, is welcome.  And if any PC-BSD people have ideas on how to 
turn things like /usr/local/etc/rc.d/pc-samba into service bundles, 
those are welcome too.  (Note that pre-supplied service bundles already 
exist for the Linux flavours of some of these, which may or may not be a 
starting point.)
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.22

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.22 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

There are several things in this release:

* a new binary package for FreeBSD
* improvements to the user-mode virtual terminal subsystem
* changes and additions to UCSPI tools
* log export to remote servers
* kqueue on Linux
* miscellany

a new binary package for FreeBSD
--------------------------------

There's now a debian-shims binary package for FreeBSD. This contains the 
heretofore not packaged invoke-rc.d and update-rc.d shim programs.  I 
decided to make this separate from the systemv-shims package because 
these areless general-purpose than those shims are.

The haltsys, fasthalt, fastpoweroff, and fastboot shims are now 
packaged, also.

improvements to the user-mode virtual terminal subsystem
--------------------------------------------------------

The console-fb-realizer now displays a mouse pointer sprite on the 
display, to reflect the position of the mouse, when an application has 
turned it on with the relevant control sequences. Mouse support via the 
evdev input subsystem on Linux is thus now fully implemented, including 
support for tablets that use absolute rather than relative positioning.

On the FreeBSD side, you can use sysmouse devices.  But this only 
permits relative positioning.  This is a limitation of sysmouse itself, 
as far as I can see.  A lot has to change, including the kernel, the 
protocol, and moused, to enable absolute positioned devices via 
sysmouse. Absolute positioning devices will therefore be supported using 
uhid devices.  Some of that is done already, but it's not complete yet.

Keyboard maps are now generated by the external configuration import 
subsystem from whatever one has in /usr/share/vt/keymaps , rather than 
being hardwired to a fixed set of countries.  In the absence of this 
directory (as will usually be the case on Linux operating systems), 
fallback U.S.A. and U.K. keyboard maps are generated.

This generation is worthy of note, as it exemplifies the mechanism that 
allows multiple BSD keyboard maps to be overlaid to make one generated 
map.  The fallback U.K. keyboard map is generated by taking the built-in 
U.S.A. keyboard map and applying a "us_to_uk" overlay map on top of it 
that only has the few differences between the U.S.A. International 
layout and the U.K. one.  (This currently produces the basic U.K. 
layout.  "U.K. Extended" should be a simple matter of another overlay 
that does the various Option+A -> a-acute mappings, but that's 
somethingfor the future.)  Similarly, versions of existing maps that 
swap Caps Lock and Control are generated by adding a simple overlay that 
does solely that.  Likewise, generated maps have an overlay applied that 
sets the Backspace key to the application-programmable DEC VT behaviour 
that console-terminal-emulator supports, that out-of-the-box BSD keymaps 
don't know anything about.

changes and additions to UCSPI tools
------------------------------------

For consistency, the UCSPI tools that supported a single --numeric 
option now support --numeric-host and --numeric-port options, for 
separately determining whether hosts and ports are taken to be names or 
just numbers.

There are now client-side tcp-socket-connect and udp-socket-connect 
tools, that open client sockets, connect them to servers, and then 
chain.  These adhere to the UCSPI conventions for inherited open file 
descriptors in client-side tools.

log export to remote servers
----------------------------

The new UCSPI clients were motivated by the new export-to-rsyslog 
command.  This is a daemon that expects to be invoked as a UCSPI client, 
connected to a remote RFC 5424/RFC 5426 ("rsyslog") server.  It 
maintains a set of "log cursors" that point to daemontools-stylelog 
directories. Tracking its position in the logs using those cursors, it 
sends new log information to the connected server.  In the usual nosh 
fashion, the filesystem is the database, and the "cursors" are just 
files and symbolic links.  The details are on the manual page.

In conjunction with the UCSPI clients, export-to-rsyslog thus makes a 
log remote export service.

This isn't intended to be the last word in such things. RFC 5426is 
unreliable, and RFC 5424 loses the microsecond and nanosecond 
information of TAI64N. But it demonstrates the idea and shows that this 
can be done in the daemontools world.  One can indeed export 
daemontools-stylelogsif one has (say) a suite of servers whose log data 
should be copied over, on the fly, to a centralized rsyslog server.  
There's room here for someone to take this idea and run further with it 
using something like RELP.

miscellany
----------

The several miscellaneous items include OpenLDAP services in the 
autoconfiguration subsystem and some tweaks to the /etc/fstab conversion 
on Linux to deal with records that don't explicitly say either 
read-write or read-only, resulting in undocumented behaviour in the 
Linux fstab parsing library.

On the subject of working around the behaviours of Linux libraries ...

libkqueue
---------

Those familiar with the development will know that Linux's libkqueue has 
been a perennial difficulty.  Its inaccessible private internal file 
descriptors are not marked close-on-exec, leaving open a security hole 
if libkqueue is used in a privileged process that forks off unprivileged 
children to run other programs.  NOTE_WRITE for EVFILT_VNODE isn't 
implemented correctly for directories.  And itbreaks when inotify events 
come through that have filenames in them.  Those familiar with the code 
will know that there was quite a lot of conditional compilation as a 
consequence, replacing libkqueue with individual hand-rolled mechanisms 
in those programs where libkqueue simply doesn't work or creates 
security weaknesses.  The final straw was a user reporting 
service-dt-scanner abending on Gentoo Linux when the scan directory is 
merely listed with "ls", which we eventually tracked down to libkqueue bugs.

No more.

I tried the route of patching libkqueue.  It was my preferred route.  
It's fairly easy to see where to add in the close-on-exec flags, for 
example.  The difficulty is in getting such things available both to 
users using Debian binary packages (on all of its various 
"stable"/"oldstable" flavours) and to users building from source on 
distributions that I don't have myself. In the end I took a step back 
and pondered whether libkqueue was even the right thing to be using in 
the first place.  After all, it's built to select from a multiplicity of 
implementations for several operating systems, using an internal 
abstraction layer,where the nosh toolset is only in fact ever using it 
for one.

So there's now an internal C++ kqueue/kevent library for Linux in the 
toolset, not ideal but "good enough" for the use that the nosh toolset 
needs from kqueue/kevent and doing the various things that it needs like 
close-on-exec, inotify with filenames and multiple events in one go, and 
proper NOTE_WRITE for directories; and the conditional code, the 
individual hand-rolled mechanisms (apart from one), and the binary 
package dependenciesfrom libkqueue are now gone.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.23

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.23 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

There is one major item in this release.

* I've adjusted console-fb-realizer's keyboard handling on BSD to use 
USB directly.

There are a few more minor changes.

* I've upgraded the version of clang++ that is used to build the binary 
packages to 3.8.0. This should have no visible effect whatsoever.  (-:
* The handling of the DECDA2 control sequence by 
console-terminal-emulator now copes with what vim sends.  (What vim 
sends isn't what my DEC VT tests had been checking.)
* convert-systemd-units now inserts uses of the ionice and chrt 
chain-loading commands on Linux into the generated service bundles.

Mostly this is a clearing the decks release in the hope that I will be 
able to do some more work on the remaining few FreeBSD conversions 
before the new year.

USB keyboard support
====================

The keyboard handling is a change to using the USB HID devices 
(/dev/uhid*) on FreeBSD in preference to (but not forcibly instead of) 
the ATA keyboard protocol.  In part this is in order to handle the 
"consumer" keys that USB has.  In part this is in order to handle the 
extra keys that one finds on 106-key, 107-key, and 109-key keyboards and 
on some numeric keypads (such as the ABNT2 thousands-separator key).  In 
part it's to remove an extra layer of the user-space virtual terminal 
system that can be outwith the kernel.  In part it's to match the USB 
mouse capability from version 1.22 of the toolset.

Please note that the structure of kbdmap files has changed slightly, to 
accomodate mappings for "consumer" keys, to reposition the entries for 
some of the 106/107/109-key keyboards' extra keys, and to cover all of 
the function key gymnastics that vim can accept.  The 
/etc/system-control/convert/ system should automatically re-convert your 
VT kbd files into the new format.As part of this, I've moved the mapping 
for the Euro symbol in the fallback U.K. layout (as generated on Linux 
in the absence of VT kbd files).  It used to be level 3 shift on the 
[eE] key in prior versions of the toolset.  Almost all real U.K. 
keyboards nowadays have it engraved as level 3 shift on the [4$] key, 
and that's where it now is.

Also note that I'm still working on this.  There might be further 
changes in 1.24.  I've found a U.K. keyboard with two [#~] keys (at A00 
and C12), and I need to check out whether this actually employs what I 
had thought to be an error in the USB HID usage tables (distinct usages 
for "\|" and "Europe1") and had corrected, or whether this is a quite 
mad keyboard that simply has two "Europe1" keys (or two "\|" keys).  
Also, I've ordered an ABNT2 and a Japanese USB keyboard, and hope to do 
some testing with them, which may prompt further tweaks.  (I really 
wanted to buy a Leadership 4530 keyboard.  They seem to be out of stock 
in a lot of places.)
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.24

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.24 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

Minor items in this release include:

* A fix for BSD keyboard layout import, that makes both "duml" and 
"ddia" be U+0308 for now.  Technically, diaeresis and umlaut are 
distinguishable in Unicode decomposed forms (using U+034F).  But for 
now, everything is simple unadorned combining diaeresis.
* A few more service bundles, for DBMail and for sudo (which in its 
vanilla form puts its timestamp files in /var/lib instead of /var/run 
and needs a cleanup service -- see Debian Bug #786555).
* Use of rtprio and idprio when converting system service units on 
FreeBSD/PC-BSD.
* Improvements to the framebuffer video mode selection in user-space 
virtual terminals for FreeBSD/PC-BSD.  It now comes up in the same 
display size as on Debian Linux on my test machines.
* Doco and other fixes from user feedback on version 1.23.  (I've 
already begun some further VirtualBox host adjustments, as we discussed, 
for 1.25.)

There is one major item in this release.

PC-BSD 10.2
===========

Until now, I'd been testing on a PC-BSD system that had been upgraded, 
with various contortions, from version 9.  This was still using UFS 
filesystems, listed in /etc/fstab; which the external configuration 
import subsystem had been happily importing to native service bundles.  
Over Christmastide I installed a PC-BSD 10.2 system from scratch, 
discovering some interesting oddities.  These included installation 
failing if you tell it that you are in the United Kingdom using a U.K. 
keyboard (PC-BSD Bug #12986); and the GRUB menu editor, as configured by 
the installer, operating in a truly eye-stretching 46 column by 28 row 
mode (by my count), and not displaying the underscore character correctly.

The important thing to know is that PC-BSD has for some time (at least 
since 2013) been ZFS-only, as far as installation goes.  (One can of 
course mount other filesystem types after installation.)  As Henry Ford 
might have said "Any customer can install to any filesystem type that xe 
likes, as long as it is ZFS.".  The result is that if installing from 
scratch one gets a whole load of ZFS datasets, and an empty (save for 
/proc and swap) /etc/fstab file.

So the major push for version 1.24 has been to get the configuration 
import system to deal with this, which it now does.  It will create 
mount services for all ZFS mounts, enable the ones that are "on", give 
them an inter-service ordering, and deal with the special-casing for the 
root (which the installer, oddly, marks as not automatically mounted, 
even though it of course is).

Alongside this, a large chunk of the remaining NetBSD rc.d services, 
from the on-going project to entirely replace them, have been crossed 
off the list.  These include mfs for /tmp, static networking and static 
ARP, pefs, serial port BPS and framing setup, ppp, rfcomm_pppd, 
persistent "entropy" for the randomness subsystem, and ipfw.  The 
progress of this work has been open from the start, and you can follow 
along on the roadmap WWW page.  Indeed, you can even join in, if you can 
convert any of the remaining few items.

There's more work to be done.  But I now have ZFS-only PC-BSD 10.2 
running nosh system-managed and service-managed.

Some notes for those eager to follow:

* Yes; I'm working on a pcdm service.  No; it doesn't help that it's 
undocumented.  Yes; that hoopla and palaver with forked subshells and 
multiple while loops calling sleep is exactly the sort of thing that 
proper service management is intended to obviate.

* If you have problems with devd, stale nologin from previous boots, and 
other things that use /var/run, it's because the convert_varrun service 
isn't enabled and your system has not been thus or otherwise migrated to 
/run.  This will be properly enabled by a preset in the next version.  
Enable it and reboot.  Or just start it and reboot. Or just boot into 
rescue mode and turn /var/run into a symbolic link to /run yourself.

* No; the nosh-run-system-manager package doesn't work properly on 
PC-BSD, as it does on vanilla FreeBSD.  PC-BSD 10.2 doesn't use the 
FreeBSD boot loader, like my old upgraded installation of PC-BSD 9 did.  
It uses GRUB.  The PC-BSD people apparently plan to get rid of GRUB in 
the future, and use the FreeBSD loader once more.  So this problem goes 
away when GRUB does.  (-:  In the meantime, use 'set 
kFreeBSD.init_path="/sbin/service-manager"' in the GRUB configuration.

* The root-resizing subsystem that was new to FreeBSD version 10 still 
needs conversion.  But ironically it doesn't work on PC-BSD 10.2 in the 
first place.  It can only grow UFS volumes, and PC-BSD's root is not a 
UFS volume.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.25

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.25 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

As you may have noticed from discussions elsewhere, a new 
oom-kill-protect utility has snuck in at the last moment.  This takes 
Linux-style OOM Killer score adjustments (an integer between -1000 and 
1000), BSD-style binary YES/NO settings, or a special setting for 
querying the "oomprotect" environment variable; and tries to do the 
closest matching thing for each platform.  Details are in the manual, of 
course.  With this, the OOMScoreAdjust setting is now converted by the 
convert-systemd-units utility.

The local-syslogd, udp-syslogd, and syslogd service bundles make use of 
oom-kill-protect with the special environment variable setting in their 
run programs.  So FreeBSD bug #204741 is addressed in a more general 
fashion that can be easily used in other service bundles. "rcctl set 
syslogd oomprotect YES" and "rcctl set syslogd oomprotect NO" can be 
used to turn OOM Killer protection on and off.

Other things in this version include:

* More configuration import utilities, covering ip6addrctl, webcamd, and 
NFS settings.
* A fix for a problem with configuration import on Linux in version 1.24.
* Two minor utilities for querying the fstab database, get-mount-what 
and get-mount-where, needed by the configuration import for mdconfig 
(but generally usable).
* New binary "run-" packages for OpenSSH server, syslog on a local 
socket, and klog.
* The new syslog and klog packages provide the Debian package manager's 
virtual package names "linux-kernel-log-daemon" and "system-log-daemon" 
(per Debian Bug #67604).

As can be seen from the roadmap, we are nearing the end of the rc.d 
conversion for FreeBSD.  Additions in this release include nfsserver, 
gptboot, rtadvd, virecover, and pcdm.  Almost all of mdconfig is 
actually done, bar some after/before orderings.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.27

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.27 .

* http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh.html
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

In fact, it is soon to be version 1.28.  This is a somewhat delayed 
notice for 1.27, because I forgot to send out the notices for versions 
1.27 and 1.26 after updating the WWW site.

As can be seen from the roadmap, we are at the point in rc.d conversion 
for FreeBSD/PC-BSD where it's actually easier to count the things that 
remain unconverted.  Discounting the PC-BSD Active Directory services 
and a handful of suspect FreeBSD services (such as growfs, which doesn't 
apply to ZFS in the first place) the remaining things to be converted 
can be counted on the fingers of one hand.  The external configuration 
import mechanism has gained the ability to handle stf, atabridge, 
mdconfig, and a few others. There are also a whole bunch more service 
bundles: cross-platform, for Linux, and for BSD.

* 
http://homepage.ntlworld.com./jonathan.deboynepollard/Softwares/nosh/roadmap.html#FreeBSDrc.d

The OOM Killer avoidance measures from version 1.25 are now employed in 
the PostgreSQL service bundle as well.

The systemd service unit conversion tool has gained a whole load of 
NUMA-related extensions: NUMAInterleave, NUMAMemBind, NUMACPUNodeBind, 
NUMAPhysCPUBind, NUMALocalAlloc, and NUMAPreferred.  These it translates 
into the equivalent invocations of the numactl chain-loading utility.

It has also gained a couple of minor fixes and tweaks.  The %m 
substitution now works, and service bundles comprising FIFOs or AF_LOCAL 
sockets are now created so that they are ordered after any relevant 
filesystem mount services.

By request, the nosh Guide has gained a whole chapter of cheatsheets, 
giving quick one-liner pointers to some common tasks. The chapter is 
divided into three sections: chain loading, logging, and service 
management.  The service management division is subdivided into 
daemontools-style commands, systemd-style commands, OpenBSD-style 
commands, SMF-style commands, and common commands. The chain loading 
division gives a number of the more common commands used in 
chain-loading run scripts (and whereever else one might want to use them).

There have been improvements in static network setup, including fixes 
for some bugs in static_arp and static_ndp and a more cross-platform 
replacement for the static-networking service.

The nosh-bundles package now supplies several aliases for services, 
which are just plain old symbolic links.  So (for example) one can 
address the CUPS service as either org.cups.cupsd or just plain cupsd.

Things to look forward to in version 1.28 already include: more service 
bundles; another chain-loading utility; a major revision to MySQL and 
MariaDB service bundling, to reflect the pushes by their own developers 
to obviate their rc scripts and the mysql-safe command and just run 
mysqld directly under service management using the tools provided by the 
service management system; and a change relevant to the all-important 
linux_logo command.  (-:
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.28

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.28 .

* https://jdebp.eu./Softwares/nosh/
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

There's a lot in this one: MySQL and MariaDB changes; more prophylaxis 
for Desktop Bus bus activation; improvements to systemd unit conversion; 
support for the old svc -x; machineenv; improvements to service 
management; fixes for the per-user manager; improvements to the console 
terminal emulator; BSD boot mode changes; the ability to pass more open 
sockets to connection-accepting programs; cron; and OpenBSD.

Italics and colour
==================

* https://jdebp.eu./Softwares/nosh/italics-in-manuals.html

This isn't a toolset change, per se.  But the WWW site now has a guide 
to seeing actual italic text in manual pages.  The nosh toolset's 
user-space virtual terminals support true italics (if one has the fonts) 
or obliquing, and this works with them.

MySQL and MariaDB changes
=========================

* https://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html

New in version 1.28 is a different and up-to-date way of managing MySQL 
and MariaDB server services — where "new" translates to finally getting 
rid of that unnecessary mysql_safe wrapper and doing things the way that 
daemontools-family toolset users have wanted to do them since the turn 
of the century.

There's a lengthy exposition on the WWW site, q.v..  The major visible 
effect is that your "mysql" or "mariadb" service is now an alias, for 
something like a "mysql@" or "mysql@01" (if you have [mysql01] in your 
my.cnf) service.  The configuration file import mechanism tries to 
construct/update mariadb@NN and mysql@NN service bundles for you, based 
upon your MariaDB and MySQL configuration files.

Further prophylaxis for Desktop Bus bus activation
==================================================

* https://jdebp.eu./Softwares/nosh/avoid-dbus-bus-activation.html

The nosh toolset now comes with a dbus-daemon-launch-helper 
replacement.  The purpose of this is to sit in your 
/usr/local/etc/dbus-1/system.conf (or equivalent) and redirect to 
service management attempts, by the Desktop Bus broker daemon, to 
demand-start services.  It is slightly fiddly to install, requiring 
manual setup by the system administrator, there being no simple way to 
add overrides to /usr/local/etc/dbus-1/system.conf and it requiring that 
you allow the "messagebus" user the necessary access for starting and 
stopping services (but not necessarily *superuser* access — rembember ACLs).

To assist with this, several popular Desktop Bus "services" now exist as 
alias names for service management services.  These are just symbolic 
links to the service bundle directories, of course. So, for example: 
With the helper in place, Desktop Bus bus activation will try to 
demand-start a service named "org.freedesktop.PackageKit" using service 
management.  This is just an alias for the "packagekit" service.

Improvements to systemd unit conversion
=======================================

Ideal mode is now closer to the daemontools-family mainstream, 
defaulting to the daemontools-family norm of always restarting 
services.  Quirks mode, conversely, now implements more of the 
non-daemontools redirection semantics for standard I/O, in particular 
with regard to listening socket units.  Some more Linuxisms have been 
added.  Limits (where applicable) can now take SI and IEC suffixes (so 
you can, say, express limits in kiloseconds).  This latter is actually 
an augmentation to the underlying softlimit command.

Passing more open sockets to connection-accepting programs
==========================================================

The improvements to systemd unit conversion also allow passing more than 
one listen()ing socket to connection-accepting programs.  You can use, 
say, ListenStream and ListenDatagram and the conversion utility will 
translate this into an appropriate chain of multiple invocations of 
udp-socket-listen and tcp-socket-listen.  It will do 
local-stream-socket-listen, local-datagram-socket-listen, 
netlink-datagram-socket-listen, and fifo-listen too.

The motivator for this was Daniel J. Bernstein's dnscache.  I have 
modified versions of tinydns, dnscache, and taiclockd that understand 
the LISTEN_FDS protocol for their being told about listening sockets 
that have been opened for them, and don't open their own sockets in that 
case.  dnscache, in particular, takes a UDP socket and a listening TCP 
socket.  The UCSPI tools in this version of the toolset can now provide 
these two to a dnscache process.  One simply chains through 
udp-socket-listen and tcp-socket-listen to dnscache, using the 
--systemd-compatibility flag.

The sharp-eyed will notice that the tinydns and dnscache services are 
following in the footsteps of the mariadb and mysql services, being 
instantiated for relevant IP addresses by the configuration import 
subsystem instead of being single-instance services.  It's not quite 
ready in this version, but you can see where things are headed.

svc -x
======

The -x option to service-control, a.k.a. svc, is no longer defunct. It 
does what unload-when-stopped does.

machineenv
==========

This was a little helper that was needed for running one particular Java 
program under nosh service management.  It's analogous to userenv, but 
instead of setting up environment variables from login information it 
sets up environment variables from machine information.

Improvements to service management
===================================

The system-control utility now respects a new "use_hangup" flag in 
service directories, as documented in service-manager's manual. This is 
used in the various getty@, agetty@, mgetty@, emergency-login@, and 
ttylogin@ service bundles, and causes system-control to attempt to bring 
down these services (at shutdown, for example) with SIGHUP, because some 
job control shells ignore SIGTERM and that delays shutdown.

Incidentally: There's now a family of pre-supplied agetty/mgetty/getty@ 
service bundles, set up for the various conventional serial terminal 
devices on OpenBSD, FreeBSD/PC-BSD, and Linux.  These are aliased from 
parallel ttylogin@ names, for consistent handling of /etc/ttys import 
and so forth.  They run a modem-aware getty, though; which of course the 
virtual terminal ttylogin@ services do not.  See the Terminals chapter 
of the nosh Guide.

Fixes for the per-user manager
==============================

The per-user manager, an instance of which you get when you run 
"system-control start user@${USER}" (or, more specifically, 
"system-control start user-services@${USER}"), is now more functional.  
The configuration importer sets you up with your own personal service 
bundle directory tree in ${HOME}/.config/service-bundles/ with a 
pre-supplied exit/ service bundle.  (Don't delete it! The per-user 
service manager needs to have a goal when it is signalled to terminate.)

Improvements to the console terminal emulator
=============================================

SoftTerm has a bugfix.  CUF and CUF no longer cause sideways scrolling.  
It also now understands some of the more esoteric control sequences from 
the old (pre-version 10) FreeBSD kernel terminal emulator, increasing 
the compatibility with programs that use the "cons25" terminal type.

The NetBSD terminal type for virtual terminals is now pcvtXX, and the 
OpenBSD one is pccon, per the 2015 termcap database.

There is also a minor fix in the SCO Console mode emulation.  All of the 
function key control sequences were off by one.  They are not, now.  (-:

BSD boot mode changes
=====================

* https://jdebp.eu./FGA/emergency-and-rescue-mode-bootstrap.html

I'd like to see FreeBSD and PC-BSD kernel loaders support the -b flag.  
In the meantime, for simple utility, I've changed the -s flag to invoke 
emergency mode rather than rescue mode, but only on FreeBSD/PC-BSD.

OpenBSD
=======

One of the servers providing the new WWW site is an OpenBSD machine, 
running the aforementioned modified versions of Bernstein tinydns and 
dnscache, and Bernstein publicfile, under nosh service management.  The 
nosh toolset now builds and runs on OpenBSD 5.9.

There are an awful lot of limitations to OpenBSD, from lack of realtime 
signals and nmount() to its older packaging tool.  Some of them are 
surmountable: I could write an nmount() shim function as I did for 
Linux, although for OpenBSD it would be a lot more complex. Some of 
them, like the lack of fexecve(), are not.  In concert, they preclude 
nosh as a system manager and packages with full automatic setup and 
teardown of basic system services.

But one can run the nosh service manager under OpenBSD rc; and 
everything else, from UCSPI servers to cron, under that.  I currently do.

I'm not particularly bothered about the package tool and the missing 
setup/teardown of the binary packages, myself.  I'm not running my 
system using installed binary packages in the first place.  I'm building 
from source, in true slashpackage style, into /package/admin/*/command/ 
and symbolically linking from /usr/local/bin/* to there.

The user-space virtual terminal system has not been tested on OpenBSD, 
and almost certainly doesn't quite work yet.  As mentioned earlier, the 
OpenBSD terminal type is set correctly by vc-get-tty. But 
console-terminal-emulator does not have an OpenBSD mode yet; 
console-fb-realizer doesn't really know how to use OpenBSD HID devices 
yet; and OpenBSD has the old 1980s-style pseudo-terminal management 
system instead of the "new" "UNIX98" 1990s one, which I have not 
tested.  At one point I was in a halfway-done position of having the 
nosh toolset's UCSPI tools but none of the service bundles.  So the 
toolset now has a tcpserver shim, that simply maps onto 
tcp-socket-listen, ucspi-socket-rules-check, and tcp-socket-accept.  It 
does not have the same defaults as the Bernstein tcpserver, though; and 
is in a separate binary package.

The cron mess
=============

OpenBSD has yet another not-quite-the-same version of cron, with yet 
another way to tell it not to uselessly fork.  So now there's an 
openbsd-cron service bundle alongside the debian-cron and vixiecron 
service bundles.  The cron alias is set up to point to the right one for 
the operating system.

debian-cron is the new name for the gnucron service bundle, by the way.  
"Debian/GNU Linux's prepackaged version" is what Ian Jackson originally 
called it in its README.  (Historical tidbit: Paul Vixie originally 
named his "PD cron".)  As far as I can tell, GNU cron was actually a 
project (by one Mike Meyer it appears) for the Free Software Foundation 
in 1987 that has entirely disappeared, if it ever existed in the first 
place.  In any case, "gnucron" is too generic a name for something that 
is at this point heading down the road of needing an individual service 
bundle for every single special flavour.  (Perhaps the Google Summer of 
Code people could add importing OpenBSD's cron's -n flag to FreeBSD cron 
to their list of ideas.  That would eliminate one special flavour and 
fix a deficiency at the same time.)

Thibault Godouet's fcron, Matt Dillon's/Jim Pryor's dcron, and Bruce 
Guenter's bcron don't need special treatment like this.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: nosh version 1.28

[Jonathan de Boyne Pollard]

I don't know why you asked about FreeBSD rc.d just on the Debian mailing 
list; but I'm going to deal in both of those and others besides, here, 
and things that apply across both, so I've re-included the FreeBSD 
mailing list.  (-:

2016-08-14 15:10, Julian Elischer:

> I don't know if I just missed it, or it isn't there but  I have a 
> question..
>
> You give examples of importing systemd service files.  What about 
> importing rc.d files with all their ability to run arbitrary shell 
> commands.
>
> And once you have the services defined, what is the logical equivalent 
> of rc.conf, which can supply parameters for each service and turn them 
> on and off?  can you import from rc.conf?
>
You did miss it.  (-:

What you missed has grown to be a significant subsystem. It was actually 
mentioned a couple of times in the 1.28 announcement. It's the external 
configuration import subsystem.  You can read about it in the nosh Guide:

xdg-open /usr/local/share/doc/nosh/external-formats.html

As you can see, there's a whole section on importing from rc.conf into 
native service management mechanisms.  ("rc.conf" covers several 
sources, note, including a FreeNAS configuration database and 
/etc/defaults/rc.conf .)

The native service mangement mechanisms are the "enable" and "disable" 
subcommands to the system-control command, and using the envdir command 
in the normal daemontools-family style way.  The enable/disable 
mechanism in "rc.conf" is treated as if it were a preset (in systemd 
nomenclature).  You tell service management to "preset" a service, and 
it will look at /etc/rc.conf and /etc/rc.conf.local (as well as some 
other preset mechanisms) to determine what to set the native 
enable/disable state to.  The user manual page for the preset subcommand 
(of system-control) explains what the preset mechanisms are in detail.

You can set up environment directories how and where you like, but 
there's a convention that is shared by the "convert-systemd-units" tool, 
the "rcctl" shim, and the external configuration import subsystem as a 
whole. This convention is an environment directory named "env" that is 
in the service directory.  The "rcctl" shim gets and sets variables 
there; and the import subsystem places converted "rc.conf", /etc/fstab, 
/etc/ttys, /etc/my.cnf, and other stuff there.

One example of this in action, out of many in the import subsystem, is 
jails that have been set up the version 9 way in "rc.conf".  Those are 
turned into service bundles, with "env" environment directories that 
contain environment settings such as "hostname", "mount_devfs", and 
"interface".  The "run" script for the jail service very simply turns 
the environment variables into arguments to the "jail" comand.  In a 
system with an original OpenBSD "rcctl" command, one would expect to be 
able to set (version 9) jail control variables by manipulating 
/etc/rc.conf with commands like "rcctl set wibble hostname wobble".  The 
"rcctl" shim and this shared convention mean that one need not stray 
that far from this if "rcctl" is one's habit: "rcctl set v9-jail@wibble 
hostname wobble" does the "native" thing of setting the "hostname" 
variable in the (conventional) environment variable directory for the 
"v9-jail@wibble" service.

Bonus feature for those with other habits: With nosh service management 
in place, one can actually import from /etc/rc.conf settings *on Debian* 
(as long as one sets up a FreeBSD/PC-BSD-style /etc/defaults/rc.conf 
pointing to it with rc_conf_files).  One can use /etc/ttys, too.

As for importing scripts that run "arbitrary shell commands", there are 
several points.

First, you may not need to.  Note that most of what you get out of the 
box in /etc/rc.d/ and /usr/local/etc/rc.d/ on FreeBSD and PC-BSD has 
already been converted.  Remember that project that I had to convert 157 
services?  Take a look at the nosh roadmap page.  It's almost done.

Second, you may not need to.  Take a look at what actually comes in the 
nosh-bundles package nowadays.  Discounting the 'cyclog@' service 
bundles there are just over 540 service bundles in there, from samba to 
ntp, from saned to ossec@agentd.  (Including the 'cyclog@' service 
bundles, it is over a thousand service bundles.)  The Debian world 
doesn't get left out, either. Although it's a lot more difficult than in 
the BSD worlds to come up with a list of "core" Debian services, a lot 
of the basics of Debian are also covered by this, from kernel-vt-setfont 
through irqbalance to update-binfmts.  And those more-than-540 service 
bundles cover lots of "non-core" stuff, from (as aforementioned) 
OSSEC-HIDS, Salt, and RabbitMQ to publicfile httpd over IPV6.

Third, you may not need to.  This was mentioned in the 1.28 
announcement, in fact.  The external configuration import subsystem 
makes *further* service bundles, beyond the pre-made ones that come in a 
binary package.  It creates service bundles to run (optional) per-user 
service management, per-user Desktop Bus brokers, MySQL and MariaDB 
servers (according to your my.cnf), PPP and SPPP, md and pefs, jails 
(set up with v9 rc.conf or the PC-BSD Warden), tinydns and dnscache 
services (not quite ready when 1.28 came out, as the announcement said), 
static IP4/IP6/ARP/NDP setup and teardown, and more besides.

Fourth, you may not need to.  Out of all of this, there's probably 
already an existing service bundle for something similar that one can 
copy and adapt.

Fifth, you may not need to.  The convert-systemd-units tool exists, 
after all.  If there's a system service/socket unit around, converting 
that may well be simpler starting point than starting with an rc 
script.  It's usually significantly simpler than starting with a van 
Smoorenburg rc script, although Mewburn and OpenBSD rc scripts can 
themselves be fairly simple starting points.  I did a "make fetch" on 
the PC-BSD ports tree a couple of months ago.  (As an aside: there are 
several broken ports that don't do the right thing here.)  There are a 
growing number of packages where there's now a systemd service/socket 
unit in the fetched source archive.

Sixth, the easy cases are easy.  As just noted, Mewburn and OpenBSD rc 
scripts can themselves be fairly simple.  (They are not *always* so, 
though, contrary to popular belief.)  If you have an rc script that says 
"The command name is this, its arguments are that.", it is very easy 
indeed to convert this into something that can execute as a "run" 
program.  Setting up all of the stuff around the "run" program for a 
complete service bundle is merely an exercise in two-line shell scripts 
(for things like "start" and "stop") and making directories and symbolic 
links (for things like the "before" and "wants" directories).

Seventh, the hard cases require a human being anyway.  Parsing a shell 
script that runs "arbitrary shell commands" would require creating what 
is essentially a full shell script interpreter, that can handle the 
Almquist, Bourne Again, and Korn shell syntaxes (because such a 
hypothetical *general-case* conversion tool would have to address van 
Smoorenburg rc scripts on Debian, Mewburn rc scripts on 
FreeBSD/PC-BSD/NetBSD and friends, and OpenBSD rc scripts) that knows 
about at least five quite different sets of "helper" commands (from 
start-stop-daemon to startproc) and that works out how an entire shell 
script translates into the actual acts of executing one or (in really 
bad cases) more services.  At this point, I defer to a human being 
*understanding what is needed* and writing one or more service bundles.  (-:

And there is, of course, scads of doco, written over the past two 
decades by many people, on how to write daemontools-family-style "run" 
scripts.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: nosh version 1.28

[Joe Nosay]

[-- Attachment #1: Type: text/plain, Size: 8305 bytes --]

Thank you very much.
And have a blessed day.

On Sun, Aug 21, 2016 at 7:04 AM, Jonathan de Boyne Pollard <
J.deBoynePollard-newsgroups@ntlworld.com> wrote:

> I don't know why you asked about FreeBSD rc.d just on the Debian mailing
> list; but I'm going to deal in both of those and others besides, here, and
> things that apply across both, so I've re-included the FreeBSD mailing
> list.  (-:
>
> 2016-08-14 15:10, Julian Elischer:
>
> I don't know if I just missed it, or it isn't there but  I have a
>> question..
>>
>> You give examples of importing systemd service files.  What about
>> importing rc.d files with all their ability to run arbitrary shell commands.
>>
>> And once you have the services defined, what is the logical equivalent of
>> rc.conf, which can supply parameters for each service and turn them on and
>> off?  can you import from rc.conf?
>>
>> You did miss it.  (-:
>
> What you missed has grown to be a significant subsystem. It was actually
> mentioned a couple of times in the 1.28 announcement. It's the external
> configuration import subsystem.  You can read about it in the nosh Guide:
>
> xdg-open /usr/local/share/doc/nosh/external-formats.html
>
> As you can see, there's a whole section on importing from rc.conf into
> native service management mechanisms.  ("rc.conf" covers several sources,
> note, including a FreeNAS configuration database and /etc/defaults/rc.conf
> .)
>
> The native service mangement mechanisms are the "enable" and "disable"
> subcommands to the system-control command, and using the envdir command in
> the normal daemontools-family style way.  The enable/disable mechanism in
> "rc.conf" is treated as if it were a preset (in systemd nomenclature).  You
> tell service management to "preset" a service, and it will look at
> /etc/rc.conf and /etc/rc.conf.local (as well as some other preset
> mechanisms) to determine what to set the native enable/disable state to.
> The user manual page for the preset subcommand (of system-control) explains
> what the preset mechanisms are in detail.
>
> You can set up environment directories how and where you like, but there's
> a convention that is shared by the "convert-systemd-units" tool, the
> "rcctl" shim, and the external configuration import subsystem as a whole.
> This convention is an environment directory named "env" that is in the
> service directory.  The "rcctl" shim gets and sets variables there; and the
> import subsystem places converted "rc.conf", /etc/fstab, /etc/ttys,
> /etc/my.cnf, and other stuff there.
>
> One example of this in action, out of many in the import subsystem, is
> jails that have been set up the version 9 way in "rc.conf".  Those are
> turned into service bundles, with "env" environment directories that
> contain environment settings such as "hostname", "mount_devfs", and
> "interface".  The "run" script for the jail service very simply turns the
> environment variables into arguments to the "jail" comand.  In a system
> with an original OpenBSD "rcctl" command, one would expect to be able to
> set (version 9) jail control variables by manipulating /etc/rc.conf with
> commands like "rcctl set wibble hostname wobble".  The "rcctl" shim and
> this shared convention mean that one need not stray that far from this if
> "rcctl" is one's habit: "rcctl set v9-jail@wibble hostname wobble" does
> the "native" thing of setting the "hostname" variable in the (conventional)
> environment variable directory for the "v9-jail@wibble" service.
>
> Bonus feature for those with other habits: With nosh service management in
> place, one can actually import from /etc/rc.conf settings *on Debian* (as
> long as one sets up a FreeBSD/PC-BSD-style /etc/defaults/rc.conf pointing
> to it with rc_conf_files).  One can use /etc/ttys, too.
>
> As for importing scripts that run "arbitrary shell commands", there are
> several points.
>
> First, you may not need to.  Note that most of what you get out of the box
> in /etc/rc.d/ and /usr/local/etc/rc.d/ on FreeBSD and PC-BSD has already
> been converted.  Remember that project that I had to convert 157 services?
> Take a look at the nosh roadmap page.  It's almost done.
>
> Second, you may not need to.  Take a look at what actually comes in the
> nosh-bundles package nowadays.  Discounting the 'cyclog@' service bundles
> there are just over 540 service bundles in there, from samba to ntp, from
> saned to ossec@agentd.  (Including the 'cyclog@' service bundles, it is
> over a thousand service bundles.)  The Debian world doesn't get left out,
> either. Although it's a lot more difficult than in the BSD worlds to come
> up with a list of "core" Debian services, a lot of the basics of Debian are
> also covered by this, from kernel-vt-setfont through irqbalance to
> update-binfmts.  And those more-than-540 service bundles cover lots of
> "non-core" stuff, from (as aforementioned) OSSEC-HIDS, Salt, and RabbitMQ
> to publicfile httpd over IPV6.
>
> Third, you may not need to.  This was mentioned in the 1.28 announcement,
> in fact.  The external configuration import subsystem makes *further*
> service bundles, beyond the pre-made ones that come in a binary package.
> It creates service bundles to run (optional) per-user service management,
> per-user Desktop Bus brokers, MySQL and MariaDB servers (according to your
> my.cnf), PPP and SPPP, md and pefs, jails (set up with v9 rc.conf or the
> PC-BSD Warden), tinydns and dnscache services (not quite ready when 1.28
> came out, as the announcement said), static IP4/IP6/ARP/NDP setup and
> teardown, and more besides.
>
> Fourth, you may not need to.  Out of all of this, there's probably already
> an existing service bundle for something similar that one can copy and
> adapt.
>
> Fifth, you may not need to.  The convert-systemd-units tool exists, after
> all.  If there's a system service/socket unit around, converting that may
> well be simpler starting point than starting with an rc script.  It's
> usually significantly simpler than starting with a van Smoorenburg rc
> script, although Mewburn and OpenBSD rc scripts can themselves be fairly
> simple starting points.  I did a "make fetch" on the PC-BSD ports tree a
> couple of months ago.  (As an aside: there are several broken ports that
> don't do the right thing here.)  There are a growing number of packages
> where there's now a systemd service/socket unit in the fetched source
> archive.
>
> Sixth, the easy cases are easy.  As just noted, Mewburn and OpenBSD rc
> scripts can themselves be fairly simple.  (They are not *always* so,
> though, contrary to popular belief.)  If you have an rc script that says
> "The command name is this, its arguments are that.", it is very easy indeed
> to convert this into something that can execute as a "run" program.
> Setting up all of the stuff around the "run" program for a complete service
> bundle is merely an exercise in two-line shell scripts (for things like
> "start" and "stop") and making directories and symbolic links (for things
> like the "before" and "wants" directories).
>
> Seventh, the hard cases require a human being anyway.  Parsing a shell
> script that runs "arbitrary shell commands" would require creating what is
> essentially a full shell script interpreter, that can handle the Almquist,
> Bourne Again, and Korn shell syntaxes (because such a hypothetical
> *general-case* conversion tool would have to address van Smoorenburg rc
> scripts on Debian, Mewburn rc scripts on FreeBSD/PC-BSD/NetBSD and friends,
> and OpenBSD rc scripts) that knows about at least five quite different sets
> of "helper" commands (from start-stop-daemon to startproc) and that works
> out how an entire shell script translates into the actual acts of executing
> one or (in really bad cases) more services.  At this point, I defer to a
> human being *understanding what is needed* and writing one or more service
> bundles.  (-:
>
> And there is, of course, scads of doco, written over the past two decades
> by many people, on how to write daemontools-family-style "run" scripts.
>
> _______________________________________________
> freebsd-hackers@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"
>

[-- Attachment #2: Type: text/html, Size: 9520 bytes --]

nosh version 1.29

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.29. * 
http://jdebp.eu./Softwares/nosh/ * 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project 
* http://jdebp.info./Softwares/nosh/ There's been a lot going on since 
version 1.28 . 2016 leap second ---------------- The TAI to UTC 
conversions know about the forthcoming leap second. service bundles 
--------------- As usual, there are several new service bundles, from 
powerd++ through zfsd to fwknopd. The new fs-servers target allows one 
to order the initialization of NFS servers before loop-to-self NFS 
mounts. The new multi-user-pre target is another ordering target that 
allows services such as the motd file updater to be ordered before TTY 
login services. The instantiated kdm@tty7 and kdm@ttyv6 services have 
been replaced with a single kdm service, with a view to dealing with 
display managers better in the future. I have some plans in this area. 
The Samba service names have been fixed. Debian calls them nmb, smb, and 
winbind; but the Samba doco and most places on the WWW call them nmbd, 
smbd, and winbindd. The latter names are used in the service bundles 
package, with aliases pointing to them from the Debian names. doco ---- 
The doco has been improved and kept up-to-date in various places, 
including correct descriptions of set-service-env and print-service-env 
after one confused user got in touch. PC-BSD is now named as TrueOS 
where the reference is not historical. code review ----------- As a 
result of some code review that was offered, std::auto_ptr is now gone 
and a rare memory corruption bug in safe_execvp() has been fixed. 
Building from scratch when one doesn't have a prior daemontools or 
freedt toolset installed also no longer hits a bug. configuration import 
improvements --------------------------------- In an effort to clear 
those last few remaining items on the nosh roadmap, a whole load of 
configuration import (pppd, sppp, rfcomm_ppp, dhclient, wpa_supplicant, 
natd, and hostapd) has been consolidated under the umbrella of 
static-networking. I plan to expand this further in 1.31, given how much 
is already in 1.30. Linux kernel VTs ---------------- Management of 
Linux kernel virtual terminals has some improvements, including setting 
UTF-8 canonical mode editing and keyboard composition modes, and 
emitting the control sequences that set up the screen saver. tai64nlocal 
changes ------------------- tai64nlocal has adopted a minor but 
important change from the BSD and GNU C libraries: before reading the 
start of a line it flushes its output. This came from trying to use it 
as a co-process in GNU awk. To prevent deadlocks, GNU awk co-processes 
need to be in what is effectively line buffered output mode even though 
their standard inputs and outputs are not terminal devices. This is now 
the case for tai64nlocal and it can be used to convert TAI64N timestamps 
as a GNU awk co-process. FreeBSD and TrueOS packaging 
---------------------------- The largest change, however, is in the 
FreeBSD/TrueOS and OpenBSD packaging. This is a change that is going to 
happen in the Debian packaging in a later version. It's partly to 
simplify the package maintenance, and partly a step towards having 
OpenBSD packages that work. A single package description is fed to both 
the new pkg tool that exists on FreeBSD/TrueOS and the old pkg tool that 
exists on OpenBSD. It's not perfect, as there are things that are easy 
with the new pkg tool that are hard with the old one; and the OpenBSD 
packages are still not fully functional. But things are better than they 
were. The OpenBSD service bundles package now almost properly sets up 
per-service user accounts and log directories, for example. 
======================================================================= 
=========== IMPORTANT UPGRADE NOTE FOR FreeBSD/TrueOS: =============== 
======================================================================= 
An important consequence of the aforementioned is that the semantics of 
the nosh-bundles package have changed. In earlier versions, the various 
nosh-run-* packages were how one set services running, except for a 
small rump set of services that were set up by the nosh-bundles package. 
This is now no longer the case. The nosh-bundles package now presets and 
starts no services at all. *All* running of services must be achieved 
with the nosh-run-* packages or some other sets of scripts and presets. 
To this end, there are now two new packages, 
nosh-run-freebsd-desktop-base and nosh-run-freebsd-server-base. These 
parallel the already existing nosh-run-trueos-desktop-base and 
nosh-run-trueos-server-base packages; except that they do not start any 
of the services that exist in TrueOS but do not exist in FreeBSD, such 
as the various pc-* services. You must install, for a working 
fully-nosh-managed system, exactly one of these four packages. If you 
are running nosh service management under Mewburn rc, you can of course 
run as many or as few services under the nosh service manager as you 
care to switch over from Mewburn rc. But if you are running a 
fully-nosh-managed system these packages will arrange to run the various 
fundamentals that one pretty much cannot do without, such as 
mounting/unmounting volumes, running devd and ldconfig, and initializing 
the PRNG.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: nosh version 1.29

[Jonathan de Boyne Pollard]

Bloody Thunderbird!  Here's that again, I hope without the surprise 
reformatting after pressing "send" this time:

The nosh package is now up to version 1.29.

* http://jdebp.eu./Softwares/nosh/
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

There's been a lot going on since version 1.28 .

2016 leap second
----------------

The TAI to UTC conversions know about the forthcoming leap second.

service bundles
---------------

As usual, there are several new service bundles, from powerd++ through 
zfsd to fwknopd.  The new fs-servers target allows one to order the 
initialization of NFS servers before loop-to-self NFS mounts.  The new 
multi-user-pre target is another ordering target that allows services 
such as the motd file updater to be ordered before TTY login services.  
The instantiated kdm@tty7 and kdm@ttyv6 services have been replaced with 
a single kdm service, with a view to dealing with display managers 
better in the future.  I have some plans in this area.

The Samba service names have been fixed.  Debian calls them nmb, smb, 
and winbind; but the Samba doco and most places on the WWW call them 
nmbd, smbd, and winbindd.  The latter names are used in the service 
bundles package, with aliases pointing to them from the Debian names.

doco
----

The doco has been improved and kept up-to-date in various places, 
including correct descriptions of set-service-env and print-service-env 
after one confused user got in touch.  PC-BSD is now named as TrueOS 
where the reference is not historical.

code review
-----------

As a result of some code review that was offered, std::auto_ptr is now 
gone and a rare memory corruption bug in safe_execvp() has been fixed.  
Building from scratch when one doesn't have a prior daemontools or 
freedt toolset installed also no longer hits a bug.

configuration import improvements
---------------------------------

In an effort to clear those last few remaining items on the nosh 
roadmap, a whole load of configuration import (pppd, sppp, rfcomm_ppp, 
dhclient, wpa_supplicant, natd, and hostapd) has been consolidated under 
the umbrella of static-networking.  I plan to expand this further in 
1.31, given how much is already in 1.30.

Linux kernel VTs
----------------

Management of Linux kernel virtual terminals has some improvements, 
including setting UTF-8 canonical mode editing and keyboard composition 
modes, and emitting the control sequences that set up the screen saver.

tai64nlocal changes
-------------------

tai64nlocal has adopted a minor but important change from the BSD and 
GNU C libraries: before reading the start of a line it flushes its 
output.  This came from trying to use it as a co-process in GNU awk.  To 
prevent deadlocks, GNU awk co-processes need to be in what is 
effectively line buffered output mode even though their standard inputs 
and outputs are not terminal devices.  This is now the case for 
tai64nlocal and it can be used to convert TAI64N timestamps as a GNU awk 
co-process.

FreeBSD and TrueOS packaging
----------------------------

The largest change, however, is in the FreeBSD/TrueOS and OpenBSD packaging.

This is a change that is going to happen in the Debian packaging in a 
later version.  It's partly to simplify the package maintenance, and 
partly a step towards having OpenBSD packages that work.  A single 
package description is fed to both the new pkg tool that exists on 
FreeBSD/TrueOS and the old pkg tool that exists on OpenBSD.  It's not 
perfect, as there are things that are easy with the new pkg tool that 
are hard with the old one; and the OpenBSD packages are still not fully 
functional.  But things are better than they were.  The OpenBSD service 
bundles package now almost properly sets up per-service user accounts 
and log directories, for example.

=======================================================================
===========  IMPORTANT UPGRADE NOTE FOR FreeBSD/TrueOS: ===============
=======================================================================

An important consequence of the aforementioned is that the semantics of 
the nosh-bundles package have changed.  In earlier versions, the various 
nosh-run-* packages were how one set services running, except for a 
small rump set of services that were set up by the nosh-bundles 
package.  This is now no longer the case.  The nosh-bundles package now 
presets and starts no services at all. *All* running of services must be 
achieved with the nosh-run-* packages or some other sets of scripts and 
presets.

To this end, there are now two new packages, 
nosh-run-freebsd-desktop-base and nosh-run-freebsd-server-base. These 
parallel the already existing nosh-run-trueos-desktop-base and 
nosh-run-trueos-server-base packages; except that they do not start any 
of the services that exist in TrueOS but do not exist in FreeBSD, such 
as the various pc-* services.

You must install, for a working fully-nosh-managed system, exactly one 
of these four packages.  If you are running nosh service management 
under Mewburn rc, you can of course run as many or as few services under 
the nosh service manager as you care to switch over from Mewburn rc.  
But if you are running a fully-nosh-managed system these packages will 
arrange to run the various fundamentals that one pretty much cannot do 
without, such as mounting/unmounting volumes, running devd and ldconfig, 
and initializing the PRNG.


_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.30

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.30 .

* http://jdebp.eu./Softwares/nosh/
* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

service bundles
---------------

As usual, there are more service bundles, including for the UWSGI 
"Emperor" and the new services in FreeBSD/TrueOS 11 such as ypldap. 
There are now services to run Sendmail in the same manner as the 
services that run exim.  Note that this is slightly different to the old 
FreeBSD division of labour.  There are individually controllable 
services for SMTP Submission, SMTP Relay, the Submission queue runner, 
and the Relay queue runner.

doco
----

The Guide has been extended with several new chapters, including a 
gazetteer of interesting directories, a chapter on log file 
post-processing, a chapter on logging security, a chapter on per-user 
service management, and some notes for individual services.  The 
commands list has moved from the blurb into the Guide, too, as it seems 
like something that an administrator might find handy to have available 
when there's no Internet connection.

* http://jdebp.eu./Softwares/nosh/guide.html

service management
------------------

There's now a hardlimit chain-loading command, analogous to softlimit.  
The convert-systemd-services utility now makes use of this and permits 
setting separate hard and soft limits, or only one or the other, with 
settings like LimitOFILE=32:128 and LimitNPROC=:infinity .

There's now a local-reaper chain-loading command, that can turn "local 
reaper" status for the current process on or off.  Have a care when 
using this, per the note on the manual page.  There is a 
LocalReaper=true extension to systemd service units for this.

netlink-datagram-socket-listen is now available on the BSDs for script 
compatibility.  It always aborts with an address family error.

There's a new hangup subcommand of system-control, equivalent to the 
existing -H option to svc .

enhancements to system-control stop/start/reset and single-shot services
------------------------------------------------------------------------

This is the first big item for 1.30 :

The start and stop subcommands of system-control now operate more 
quickly.  Instead of polling once per second, they monitor the 
supervise/status files of each service that is in the process of being 
started and stopped, with kevent().

In addition, system-control now supports the notion of services that 
become ready when their main process has exited, marked with a new flag 
file in the service directory.  convert-systemd-units has been modified 
to convert "oneshot" services to this, instead of to services that put 
all of the run code into the start program.  Thus "oneshot" services 
that are running their actual main programs are reported as "running" by 
svstat, rather than as "starting".

This takes advantage of the extended status information that 
service-manager has been writing to the status file since version 1.28.  
The sharp-eyed may have noticed that in version 1.28 the output of 
"svstat"/"system-control status" gained information about the exit 
statuses of the start, run, restart, and stop programs. This is what 
system-control now uses to detect whether ready-after-run services ran 
before they stopped.  (Detection of ready-after-run services that are 
running with no processes, because they are "remain" services, can be 
and is done with just the daemontools-encore-compatible status information.)

Old-style "oneshot"s will continue to work as before, as of course they 
become ready as soon as the run process is spawned, which is after they 
have run their programs as part of start.

The benefit of this new style, apart from reporting a running service as 
actually "running", which should help with nagios monitoring and the 
like, is that "oneshot" services converted from systemd no longer have 
to be marked as RemainAfterExit=true in order to avoid a dummy "pause" 
process hanging around.  This is the case for old-style "oneshot" 
services.  They have to run something in run, after all, and that 
something has to keep running in order for the service to be considered 
ready and services ordered after it to be unblocked.  A ready-after-run 
service, however, unblocks ordered-after services if it has reached the 
stopped state via a run, thus puts its programs in run, thus doesn't 
have to have a dummy pause process, and can be RemainAfterExit=false 
without adding to the process list.

log file management
-------------------

export-to-rsyslog had a bug that caused it to skip old log files (the 
@nnnnnnnnnnnnnnnnnnnnnnnn.s ones) in catch-up mode.  This has been 
corrected.  There is now a follow-log-directories command that can 
substitute for tail -F .  It knows the actual structure of log 
directories, operates using one or more cursors like export-to-rsyslog 
does, and copes correctly with cyclog/multilog log rotation (which GNU 
tail, at least, apparently has problems with when the timing is 
particularly wrong on a loaded system).

See also http://jdebp.eu./FGA/do-not-use-logrotate.html

build
-----

More warnings are now turned on with clang++ during the build, and a lot 
of the resultant warnings have been eliminated where appropriate.  The 
check for eg++ in preference to g++ is now limited to OpenBSD, where (at 
least on OpenBSD 5.9) eg++ is still ahead of g++ by a wide margin.

Per-user service management
---------------------------

Changes in per-user service management are the second big item for 1.30 :

The per-user service manager instances are now invoked via userenv, so 
all per-user services that you run under nosh service management, D-BUS 
servers or otherwise, will have your own HOME, SHELL, and USER set.  
Several per-user daemon softwares were expecting HOME to be set.

To match what the Desktop Bus people are doing, the dbus socket path for 
the per-user D-BUS broker has changed from 
"/run/user/$USER/dbus/user_bus_socket" to "/run/user/$USER/bus".  In 
theory, this is addressable (in D-BUS speak) as "unix:runtime=yes". In 
practice, there is no version of D-BUS available on stable/release 
FreeBSD, TrueOS, or Debian that understands this address syntax.  So one 
still has to use "unix:path=/run/user/$USER/bus".

The Desktop Bus people and the desktop environments people are also 
switching from per-login D-BUS brokers to per-user D-BUS brokers. The 
nosh toolset has already had this for over a year, since the middle of 
2015.  Each real-person user account has an optional per-user service 
management service (e.g. user-services@fred).  What is new is that 
per-user service bundle areas are now populated with a whole load of 
service bundles for real services, many relating to GUI desktop 
environments, and the per-user D-BUS broker has moved to there, from 
being a system-level service bundle.

The configuration import subsystem creates these new per-user service 
bundles in the home directories of individual real users, under 
~fred/.config/service-bundles/services/ and 
~fred/.config/service-bundles/targets/ (for user fred).  These run 
per-user services for a whole load of things, from GNOME editor and 
emacs through dconf and KDE Notify to urxvtd and GNOME Terminal.

The configuration import subsystem also sets up a bypass for D-BUS's 
broken "bus activation" mechanism, so that instead of attempting to run 
these D-BUS servers directly, the D-BUS broker instead tells the nosh 
per-user service manager to run them.  This takes the form of a 
replacement dbus-daemon-launch-helper, and the per-user D-BUS brokers 
now employ a modified configuration file that invokes it.

There's a full explanation of how this all works in the new chapter on 
demand-starting user-level Desktop Bus services in the nosh Guide.

Notes:
   * For emacs as a per-user service, you must have a very recent emacs 
with its very-late-to-the-party --new-daemon option.
   * GNOME Weather and its interaction with GeoClue2 are only partly 
tested, because the versions of them available for the test platforms 
were attempting to contact a weather service that the U.S. Government 
discontinued in June 2016; and this was hardwired into their code.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.31

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.31 .

* http://jdebp.eu./Softwares/nosh/

* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

* http://jdebp.info./Softwares/nosh/

This release fixes a problem with emergency mode that was introduced by 
accident in 1.29 .  The emergency-login@console service was not properly 
enabled by package installation.  Now it once again is.

There are a number of bug fixes in this release, such as rare corner 
cases in how convert-systemd-units generates arguments to pass to sh, 
what port the nginx server part of Appcafe binds to when not the 
default, the use of setuidgid-fromenv to set more than 1 supplementary 
group ID, and making the Makefile in tinydns@* services work with both 
BSD and GNU make.  Various service bundles that perform 
clean-up-directories actions at bootstrap have been made more difficult 
to accidentally re-trigger after bootstrap.

There is also a fair amount of new features:

* The automatically-generated data for tinydns@* services now 
encompasses all of the reverse lookup domain names for private/local IP 
addresses, so none of the DNS traffic involving such lookups will leak 
out of your machine/organization to the rest of Internet.

* The userenv command has gained the ability to (optionally) set a whole 
lot more environment variables from the capabilities in /etc/login.conf 
and ~/.login_conf .  It now can be used as the 
setup-the-user-environment part of a command chain that is designed to 
perform the setup of an interactive login session. This is particularly 
useful for fixing PCDM, the display manager in TrueOS.

* The pipe command can now arrange to clean up the child process in one 
of two ways.  This is made use of in the dnscache service bundles, and 
dnscache services no longer contain the perpetual zombie process that 
they had in version 1.30 .

* Presets now support wildmat-style character set wildcards. e.g. one 
can now write "ttylogin@vc[0-9]-tty" as a service name pattern.

* If you have been using the --verbose option to the start/stop/reset 
subcommands of system-control, you'll notice that it now colourizes its 
output.  Its output has also been adjusted to more clearly indicate 
blocked services and what they are blocked by.

The big item is that there is now a complete set of simple control 
groups manipulation commands, the pre-supplied service bundles all make 
use of it, and all service bundles created by convert-systemd-units make 
use of it.  (All of this is a no-op on FreeBSD/TrueOS and OpenBSD, of 
course.)

If you've read the Linux doco, you'll know that control groups do not 
require any sort of centralized gatekeeper process, and are a 
decentralized system that can be driven with just the echo command.  In 
practice, using echo is non-trivial.  The move-to-control-group, 
delegate-control-group-to, and set-control-group-knob commands take the 
hassle out of working out exactly what to echo where.  They do all of 
the hard work of determining what the directory name of the current 
control group under /sys/fs/cgroup is, and present a simple system 
allowing one to create and navigate to another control group, delegate 
control over the current control group (and its subgroups) to an 
unprivileged user, and set control group knobs.

The set-control-group-knob utility further illustrates the convenience 
functionality over and above a simple echo command. It can calculate a 
knob setting as a percentage of another number, handle SI and IEEE/IEC 
multiplier suffixes, and translate the device file names that are 
(comparatively) convenient for humans into the literal major and minor 
device numbers that the Linux control groups API actually operates in 
terms of.

There are new chapters in the Guide covering the automatic import of 
FreeBSD 9 and PC-BSD Warden jails, how jailing services on 
FreeBSD/TrueOS works, and limiting services.  The limiting services 
chapter covers both the original Unix resource limits system and Linux 
control groups.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.32

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.32 .

* http://jdebp.eu./Softwares/nosh/

* 
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

* http://jdebp.info./Softwares/nosh/

This release fixes two problems with Gentoo Linux (control group version 
detection and a problem with mounting API filesystems) that we hashed 
out on the Supervision mailing list.  It furthermore contains a change 
to the way that convert-systemd-units generates service bundles that 
fixes problems with control group setup when the service unit defines a 
"slice" for the service or when the service unit is a template. In 
furtherance of that there's a new create-control-group command.

Other things in this release include improvements to the (unpackaged) Z 
Shell command-line completions, which now display option completion 
menus properly; some improvements to the Terminals chapter in the Guide; 
fixes to various service bundles that were using shell reserved words 
and operators such as "for" and "&&" without explicitly invoking the 
shell; additions to userenv for setting DBus and XDG Runtime variables; 
and a fix that prevents "system-control reset" from looping indefinitely 
when run by an unprivileged user such as "messagebus" that lacks access 
to the control/status API.

The major improvement in this release, though, is to console-fb-realizer 
on TrueOS.

FreeBSD gives console-fb-realizer uhid device files to use for input 
devices, which speak the USB HID report protocol and which 
console-fb-realizer has been happy with for a long time.  TrueOS 
provides either ums/ukbd devices, which lack various features because 
they speak the old sysmouse and atkbd protocols, or ugen devices.  There 
are no uhid devices available. console-fb-realizer can now use the ugen 
devices.  Moreover, it will detach the ums/ukbd drivers from the ugen 
devices using the new detach-kernel-usb-driver command, so that there 
aren't two things both attempting to read HID reports.

console-fb-realizer also now correctly sets the keyboard LEDs on both 
FreeBSD and TrueOS.

There have been several minor adjustments to the kernel VT sharing parts 
of console-fb-realizer, preparatory to splitting the program up into 
separate parts for input and output devices, permitting things such as 
multiple keyboards each with its own keyboard map and numlock semantics, 
in a future release.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.33

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.33 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

This has been held back because of work being done by someone else.  I don't
want to steal xyr thunder, so I'll leave the announcement of that work to xem.
 Suffice it to say that it will interest a new group of people.

There are several major improvements in 1.33 .

Packaging
---------

In the version 1.29 announcement I said that the Debian packaging system was
going to be brought into line with the system used for FreeBSD/TrueOS and
OpenBSD.  This is now done.  Debian and the BSDs all now use a similar system
for generating each package manager's package maintenance instructions from an
abstract package description.

==============================================================
=========== IMPORTANT UPGRADE NOTE FOR Debian: ===============
==============================================================

An important consequence of the aforementioned is that the semantics of the
nosh-bundles package have changed. In earlier versions, the various nosh-run-*
packages were how one set services running, except for a small rump set of
services that were set up by the nosh-bundles package.

This is now no longer the case. The nosh-bundles package now presets and starts
no services at all. *All* running of services must be achieved with the
nosh-run-* packages or some other sets of scripts and presets.

To this end, there are now two new packages, nosh-run-debian-desktop-base and
nosh-run-debian-server-base. These parallel the
nosh-run-{freebsd,trueos}-{desktop,server}-base packages already available since
1.29 for FreeBSD/TrueOS. You must install, for a working fully-nosh-managed
system, exactly one of the nosh-run-debian-{desktop,server}-base packages.

If you are running nosh service management under systemd, you can of course run
as many or as few services under the nosh service manager as you care to switch
over from systemd. But if you are running a fully-nosh-managed system these
packages will arrange to run the various fundamentals that one pretty much
cannot do without, such as mounting/unmounting volumes, running
udev/eudev/vdev/mdev, binfmt loading, and initializing the PRNG.

Log service account names
-------------------------

The naming scheme used for the user accounts for dedicated log service users has
changed.  Installing the new nosh-bundles package should automatically rename
all existing log service accounts to use the new scheme.

The new naming scheme is slightly more compact, and copes better with services
that have things like underscores and plus characters (e.g. powerd++) in their
names.

As an ancillary to this, system-control now has an "escape" subcommand which can
be (and indeed is) used in scripts to perform the escaping transformations.

More packages
-------------

There are now four more -shims packages, for commands whose names conflict with
commands from other packages: nosh-kbd-shims, nosh-bsd-shims, nosh-core-shims,
and nosh-execline-shims.

nosh-kbd-shims, for example, contains a chvt shim that is an alias for the (also
new) console-multiplexor-control command; with it, and suitable privileges to
access the virtual terminal's input queue, one can switch between multiplexed
user-space virtual terminals in much the same way as the old chvt command does
with kernel virtual terminals.

The Z Shell command-line completion for the various commands in the toolset
(system-control, svcadm, shutdown, svstat, and so forth), which has been
available to the people building from source for a while, is now also available
as a binary package.

Configuration import
--------------------

ldconfig on TrueOS is now properly handled.  In particular, the external
configuration import subsystem now correctly pulls in and converts all of the
ldconfig directories.  (TrueOS has a lot more things that require ldconfig
support than stock FreeBSD does.)

The configuration import subsystem also now handles instances of Percona server,
alongside MySQL and MariaDB.  Moreover, these are now handled by the same set of
service bundles, which always produce service bundles named mysql@*.  MySQL
version 5.7 or later is now assumed.

The configuration import subsystem now automatically generates OpenVPN service
bundles based upon the current OpenVPN configuration.

=======================
==== CAVE: OpenVPN ====
=======================

The upgrade process attempts to remove the old hardwired openvpn@server and
openvpn@client service bundles.  However, you might encounter remnants of these
service bundles lying around in /var/sv that you will find that you need to
clean up by hand.

GOPHER
------

To accompany the new gopherd server in djbwares 5, there is a gopher6d service
bundle that runs it, serving up the same static files area as http6d, https6d,
and ftp4d do.

The FreeBSD, OpenBSD, and Debian package repositories can now be browsed with
GOPHER.  This is gopherd in action.  On the server side, generating the
index.gopher files is a fairly humdrum exercise in the use of redo (to
regenerate the indexes only when the directory contents change) and printf (to
construct the GOPHER format menus).

UCSPI-UNIX
----------

Two new UCSPI tools have been added to enable UCSPI-UNIX servers to listen on
and accept connections on AF_UNIX sequential packet sockets.  udevd is one such
server, and it is now handed its listening socket at startup rather than
expected to open its own.

nosh version 1.34

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.34 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

Once again, there are a few more service bundles.  The most interesting ones in
this version are perhaps the finish-update and finish-install targets, designed
to be invoked the first bootstrap after an update or install has been done, and
the users target, which is used to auto-start per-user subsystems at bootstrap.
 Several NFS service bundles are now common across operating systems.  And the
OpenVPN service bundles are now split into separate client and server services.

Several minor bugs have been fixed here and there: a duplicated newline in
line-banner that was throwing off publicfile FTP service; a problem with
recordio on FreeBSD/TrueOS; and a problem with attempts to use slashes in
environment variables in service bundle environment directories.

The user-space virtual terminal emulator now implements the Xterm extensions to
DECSCUSR, and the framebuffer realizer can display the resultant cursor shapes.
 This can be made use of by programs such as Neovim.

There are now separate service bundles and nosh-run- packages for running eudev
and systemd-udev, because the two are now significantly divergent.

The various utilities for changing the process environment no longer use the GNU
C library/BSD C library functions for doing so, and so no longer suffer from the
concomitant memory leaks that their manual pages used to warn about.

The convert-systemd-units tool has been slightly enhanced, for the benefit of a
fix that has been made to the per-user gpg-agent service.

The external configuration imports system has been extended.  It now deals with
importing the hostname configuration value, taking that responsibility away from
and simplifying the set-dynamic-hostname utility.  It now imports various Debian
and other kernel virtual terminal settings, from /etc/kbd/config,
/etc/default/console-setup, and /etc/vconsole.conf .  And network configuration
import now can set up services for both dhcpcd and dhclient.
 /etc/system-control/convert/rc.conf now contains more settings on Linux
operating systems as a result, including dhclient_program.
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

nosh version 1.35

[Jonathan de Boyne Pollard]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

The nosh package is now up to version 1.35 .

  *

    http://jdebp.eu./Softwares/nosh/

  *

    https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

  *

    http://jdebp.info./Softwares/nosh/


    Networking

As I mentioned a week or so ago, the external configuration import 
subsystem now converts a Debian-style /etc/network/interfaces 
configuration file, via rc.conf settings, into the native networking 
subsystem.

There is also a whole new /Networking/ chapter in the /nosh Guide/, 
which explains this and several other things, including how Plug and 
Play integration interoperates with the networking services and what the 
native networking subsystem encompasses, to the level of what service 
does what and to what purpose.

Work on the Plug and Play integration is on-going, and I hope to have 
yet more for this, and indeed for other parts of the networking 
subsystem, in version 1.36.


    Packages

There are some Debian packages that declare that they need the logrotate 
package, even though they do not when run under nosh service 
management.  For their benefit there is now a nosh-logrotate-shims 
Debian package that is simply a dummy package that satisfies this need 
without setting up a spurious and unnecessary logrotate system.


    Service bundles

There are a few more service bundles, including ones for sysstat and 
elasticsearch.  The existing service bundles for things such as unbound, 
clamav, and freshclam have been augmented and fixed in response to user 
feedback.  And a bug that incorrectly resulted in the ldconfig service 
being disabled has been fixed.

The dbus services, the system-wide one and the per-user one(s), have 
been renamed to dbus-daemon.  This is because of the existence of a 
dbus-broker service bundle.  This is a placeholder for if the 
dbus-broker people ever fix it so that it works. dbus-broker does not 
provide a working system right now.  It is currently not possible to 
substitute dbus-broker for dbus-daemon on non-systemd systems, because 
dbus-broker is very tightly tied in to systemd's idiosyncratic D-Bus 
control interface.  It /only/ speaks the systemd-specific protocol, and 
knows no other way of stopping and starting services, not even the 
service command.  (In contrast dbus-daemon can still be configured to 
demand-start services using simple service management commands 
<http://jdebp.eu./Softwares/nosh/avoid-dbus-bus-activation.html#ChangeDBusHelper>.)


[-- Attachment #2: Type: text/html, Size: 3416 bytes --]

nosh version 1.36

[Jonathan de Boyne Pollard]

[-- Attachment #1: Type: text/plain, Size: 2872 bytes --]

The nosh package is now up to version 1.36 .

  *

    http://jdebp.eu./Softwares/nosh/

  *

    https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

  *

    http://jdebp.info./Softwares/nosh/


    More Java tools

This release comes with the |find-default-jvm| and |find-matching-jvm| 
tools, which will set up the |JAVA_HOME| environment variable to point 
to a default/matching JVM directory, using the FreeBSD/TrueOS and Debian 
conventions for locating JVM directories.  To match these, 
|convert-systemd-units| now recognizes |JVMDefault|, |JVMVersion|, 
|JVMOperatingSystem|, and |JVMManufacturer| extensions to the systemd 
unit file format.


    Tool improvements

|convert-systemd-units| now recognizes a |MachineEnvironment| extension 
to the systemd unit file format, which controls the generation of an 
invocation of |machineenv|.  It also now recognizes and translates 
|RDMAHCAHandlesMax| and |RDMAHCAObjectsMax| settings.

The |unshare| command now has flags for specifying process ID and user 
ID namespaces on Linux.

The |setup-machine-id| command now correctly falls back to the old D-Bus 
files on FreeBSD, which it had not been doing because of a bug.


    New system management features

In support of an initiative by Warner Losh, there is support for power 
cycling via hardware and a kernel that support it.

The system manager treats |SIGRTMIN+6|, unused in the systemd system, as 
a request to invoke a new |powercycle| service bundle; and 
|SIGRTMIN+16|, similarly unused, as the underlying actual powercycle 
request, which it translates to either |RB_POWERCYCLE| if it is present 
in the C library headers, or |RB_AUTOBOOT| if it is not. There is a new 
|system-control powercycle| subcommand, which defaults to sending these 
signals.

Note that the binary packages are currently built on a system that lacks 
|RB_POWERCYCLE| in the C library.

The compatibility |shutdown|, |reboot|, |halt|, and |poweroff| commands 
all now sport a new |-c|/|--powercycle| option.  There are new 
|fastpowercycle| and |powercycle| commands.  The |system-control init| 
subcommand now sports a new |c|/|C| argument, by analogy to |h|/|H|. And 
this is of course thus reflected automatically in the compatibility 
|telinit| command and the |initctl-read| server.


    Service bundles

Fixing an oversight in 1.35, the per-user |dbus| services are now 
renamed to |dbus-daemon| too.

There are a few more service bundles, including ones for |jenkins|, 
|apacheds|, |udisks2|, and |ndppd|.

The |linux-utmp| service bundle has been retired, in favour of a unified 
|utx| service bundle, which was previously FreeBSD-only, that operates 
across platforms.  In support of this, there is a new 
|login-update-utmpx| command, and a new |freebsd-shims| package that 
aliases that to the |utx| command on non-FreeBSD platforms.

nosh version 1.37

[Jonathan de Boyne Pollard via freebsd-hackers]

The nosh package is now up to version 1.37 .

  *

    http://jdebp.eu./Softwares/nosh/

  *

    https://www.freebsd.org/news/status/report-2017-07-2017-09.html#The-nosh-Project

  *

    http://jdebp.info./Softwares/nosh/

Some of the changes in this release are works in progress, that you will 
see fully realized in version 1.38 or later.

Changes include:

  *

    There is a new chapter in the /nosh Guide/ for those wishing to make
    packages and ports of other softwares, or add service bundle support
    to existing packages and ports.

  *

    The external formats configuration import subsystem has been
    reorganized a bit.

      o

        Nothing uses the |JAVA_HOME| import system any more, where
        service bundles explicitly have their |JAVA_||HOME| variables
        set by configuration import, although it is retained.  All
        service bundles instead use the |find-matching-jvm| mechanism to
        auto-detect a JVM matching their chosen criteria at start time.

      o

        The per-user services import is now in two parts. System-wide
        import sets up a |$HOME/.config/service-bundles/convert/|
        subdirectory for each (real user) user account; and each user
        can then use that, which contains a subordinate per-user
        configuration import mechanism, to set up imported per-user
        service bundles for things.

      o Per-user service source files for Desktop Bus and other services
        are now in their own subdirectory, as are converted keyboard
        maps for the userspace virtual terminals.
  *

    |static-networking| external format configuration import has been
    enhanced to set up |snort@/interface/| services and to handle
    |ipv6_cpe_wanif| and |ipv6_activate_all_interfaces| from |/etc/rc.conf|.

  *

    There is a new |make-read-only-fs| chain loading tool that is a
    placeholder for now.  It is used in some service bundles generated
    by the |convert-systemd-units| tool, which now recognizes and
    converts |CPUAffinity|, |ProtectHome|, |ProtectSystem|,
    |ReadWriteDirectories|, |ReadOnlyPaths|, and |InaccessiblePaths|
    settings.

  *

    Per-user management has been augmented, finally fixing the problem
    of |system-control| locating the per-user manager by giving the
    per-user manager an optional listening FIFO open file descriptor,
    which it uses to listen for user-wide state change commands.
    |system-control --user| |halt|/|normal|/|sysinit|/&c. now send
    commands via this FIFO, and each user's |user-services@/username/|
    service bundle now uses |fifo-listen| to set up the FIFO and creates
    the |per-user-manager/| subdirectory in |/run/user|.

  *

    There are some more service bundles in the collection that comes
    with the toolset: clickhouse-server, hue, udhcpc-log, minissdpd,
    rtkit-daemon, accounts-daemon, gdm3, speech-dispatcher, gdomap,
    blueman-mechanism, and sysvipc.

  *

    The per-user configuration import now recognizes and sets up
    per-user service bundles for a whole lot more per-user services.

  *

    On FreeBSD/TrueOS systems |setup-machine-id| now writes
    |/usr/local/etc/machine-id|.

  *

    The userspace virtual terminal services, the multiplexor and the
    terminal emulators, no longer run under the aegis of the |daemon|
    system account.  Rather, they now have their own dedicated accounts
    under whose aegides they run.  To go with that, there is now a
    |user-vt-realizer| group to which users can be added to grant them
    realizer (i.e. front-end I/O) access to the system-wide userspace
    virtual terminals.

  *

    A common build problem across several toolsets that occurs if one
    has set a |CDPATH|, has been fixed.  Various tweaks have also been
    made to make life easier for Archnosh and ports to other operating
    systems.

_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"

Re: nosh version 1.37

[Guillermo]

2018-02-18 3:49 GMT-03:00 Jonathan de Boyne Pollard:
>
> The nosh package is now up to version 1.37 .
> [...]
>    Per-user management has been augmented, finally fixing the problem
>    of |system-control| locating the per-user manager by giving the
>    per-user manager an optional listening FIFO open file descriptor,
>    which it uses to listen for user-wide state change commands.
>    |system-control --user| |halt|/|normal|/|sysinit|/&c. now send
>    commands via this FIFO, and each user's |user-services@/username/|
>    service bundle now uses |fifo-listen| to set up the FIFO and creates
>    the |per-user-manager/| subdirectory in |/run/user|.

\O/ \O/ \O/

On the other hand, all those new .do scripts that generate systemd
unit files and configuration files using the read_os shell function
fail on Gentoo :-P

redo-ifchange[2]: ERROR: services/dbus-broker.service: Not done.
redo-ifchange[2]: ERROR: services/dbus-daemon.service: Not done.
redo-ifchange[2]: ERROR: services/system-wide.conf: Not done.
redo-ifchange[2]: ERROR: systemd/service-manager.socket: Not done.
redo-ifchange[2]: ERROR: convert/per-user/at-spi-dbus-bus.service: Not done.
redo-ifchange[2]: ERROR: convert/per-user/gconfd.service: Not done.
redo-ifchange[2]: ERROR: convert/per-user/per-user.conf: Not done.

'read_os ID' returns 'gentoo' for Gentoo's /etc/os-release, and
'read_os VERSION_ID' returns nothing (it is a rolling release
distribution), so this always matches the

*)       ext=who ;;

lines, making the redo-ifchange invocation fail with either "Don't
know what to use to build this" or "Cannot find .do file to use". Or
making it call convert/per-user/default.do and *then* failing. So what
do I do, should I patch the .do scripts to include a 'gentoo:*)' line?

This is going to happen for every [GNU/]Linux distribution that is not
Debian, Arch, CentOS or RHEL. It does not... uh... look very portable
:/

Additionally, the convert/per-user/*.do scripts' 'read_os' function
calls 'exec' via absolute path /bin/exec instead of relative path
../../exec, which is not going to work if nosh isn't already installed
(chicken and egg). On my computer that results in accidentally calling
execline's exec program, which is even funnier.

Thanks for your attention.
G.

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:

> This is going to happen for every [GNU/]Linux distribution that is 
> not  Debian, Arch, CentOS or RHEL. It does not... uh... look very 
> portable :/
>

That is a reflection of reality.  It isn't very portable.  But you are 
misidentifying what it is.  It is the location of these various off-PATH 
executables, which have at least four different possible locations; and 
the name of the user that runs the system-wide Desktop Bus broker.  Yes, 
the people who have packaged these up don't agree on the same off-PATH 
locations; and their names are not very portable; and they don't agree 
on system account names.  If you can work out which of the existing four 
Gentoo's packaging for things like at-spi-bus-launcher is most like, 
like M. Caravia did for Arch Linux, I'll put in some mappings for 
Gentoo.  But do not blame the messenger for the news that GNOME confd is 
in four different places across Arch Linux, RedHat Linux, Debian/Ubuntu 
Linux, and FreeBSD/TrueOS.

You might find that Gentoo mainly or wholly follows the pattern of one 
of the others.  But from mine and M. Caravia's experience of operating 
systems not even being entirely *self* consistent in this area, with no 
overall pattern being followed even for Desktop Bus softwares on a 
single operating system, you probably shouldn't raise your hopes about 
that.  (-:

And yes, M. Caravia originally worked these out as a set of patches for 
Arch Linux, working out what should happen for arch:* .

Guillermo:

> Additionally, the convert/per-user/*.do scripts' 'read_os' function 
> calls 'exec' via absolute path /bin/exec instead of relative path 
> ../../exec, which is not going to work if nosh isn't already installed 
> (chicken and egg).
>

There's no chicken and egg problem.  The external formats configuration 
import requires (amongst others) the nosh-exec package.  There is no 
requirement in the opposite direction, so simply install nosh-exec 
first.  This happens automatically with the provided packaging for 
FreeBSD/TrueOS and Debian, and with M. Caravia's Archnosh packaging, 
because of nosh-bundles depending from nosh-exec.  If one installs the 
former, the package managers will have made sure that the latter is 
installed too.

Moreover that is not the relative path from your 
$HOME/.config/service-bundles/convert/ directory to /bin .  Not that 
there of course *is* a stable single relative path for such a thing, 
given that home directories can be anywhere from /export/home/guillermo 
to /var/lib/mysql .  Nor that, as mentioned, such a relative path is 
needed, given that one installs the toolset before running the external 
configuration import.

Re: nosh version 1.37

[Guillermo]

2018-02-18 20:05 GMT-03:00 Jonathan de Boyne Pollard:
>
> Guillermo:
>
>> This is going to happen for every [GNU/]Linux distribution that is not
>> Debian, Arch, CentOS or RHEL. It does not... uh... look very portable :/
>>
>
> That is a reflection of reality.  It isn't very portable.  But you are
> misidentifying what it is.
> [...]
> Guillermo:
>
>> Additionally, the convert/per-user/*.do scripts' 'read_os' function calls
>> 'exec' via absolute path /bin/exec instead of relative path ../../exec,
>> which is not going to work if nosh isn't already installed (chicken and
>> egg).
>>
>
> There's no chicken and egg problem.  The external formats configuration
> import requires (amongst others) the nosh-exec package.
> [...]
> Moreover that is not the relative path from your
> $HOME/.config/service-bundles/convert/ directory to /bin.  Not that there
> of course *is* a stable single relative path for such a thing, given that
> home directories can be anywhere from /export/home/guillermo to
> /var/lib/mysql .  Nor that, as mentioned, such a relative path is needed,
> given that one installs the toolset before running the external
> configuration import.

But wait. You are talking here about run-time requirements / behaviour
of the configuration import subsystem and the service bundles provided
by nosh, after one installs (some equivalent of) the nosh-bundles or
nosh-run-via-systemd packages for the target operating system. I
accept that OS-specific patches might be required here and there to
make them work for the reasons you mentioned.

But the 7 .do scripts I talked about are executed (indirectly) by the
package/compile script from the source package, and they just create
symbolic links. To be precise, they are executed by the source/all.do
script, because they are named in 'echo' commands that feed their
output to the 'xargs -r redo-ifchange' invocation. As it is, on any
[GNU/]Linux distribution that is not Debian, Arch, CentOS or RHEL,
downloading the source package and building it slashpackage-style by
executing package/compile will simply fail.

There are warnings in the jdebp.eu webpage about package/stage, but my
expectation was that package/compile would just work (even without a
previous nosh installation), as it has for older versions, provided
the required dependencies are installed (meaning redo, xmlto, ncurses,
pax, etc.). I can work around this by just patching source/all.do so
that it does not execute those .do scripts, or look at what they
should do for Gentoo and patch them, or whatever, but I am sure quite
a few people besides me are going to be surprised by package/compile
failing for them.

(You are right about relative path ../../exec being wrong, it should
be relative to redo's working directory, i.e. ./exec)

G.

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:
> But the 7 .do scripts I talked about are executed (indirectly) by the 
> package/compile script from the source package, and they just create 
> symbolic links. To be precise, they are executed by the source/all.do 
> script, because they are named in 'echo' commands that feed their 
> output to the 'xargs -r redo-ifchange' invocation.

Hmmm.  You're right.  Maybe that should be deferred.  I shall think 
about this.

As the release notes said, some of these are works in progress.  I was 
expecting hiccoughs in the reorganized external config import, which I 
had to rush to meet the deadline.  (I am also expecting that we have 
have to work out what the non-Debian equivalents for dpkg-divert are, if 
any.  It turns out that dbus-daemon has got worse, which has had 
knock-on effects.)  Get me those Gentoo equivalences, and I can tweak 
this stuff for 1.38.

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:
> redo-ifchange[2]: ERROR: services/dbus-broker.service: Not done.
> redo-ifchange[2]: ERROR: services/dbus-daemon.service: Not done.
> redo-ifchange[2]: ERROR: services/system-wide.conf: Not done.
> redo-ifchange[2]: ERROR: systemd/service-manager.socket: Not done.
> redo-ifchange[2]: ERROR: convert/per-user/at-spi-dbus-bus.service: Not 
> done.
> redo-ifchange[2]: ERROR: convert/per-user/gconfd.service: Not done.
> redo-ifchange[2]: ERROR: convert/per-user/per-user.conf: Not done. 

A snapshot of the 1.38 development is in the usual place.  See how far 
that gets you.

Re: nosh version 1.37

[Guillermo]

2018-02-20 16:13 GMT-03:00 Jonathan de Boyne Pollard:
>
> Guillermo:
>>
>> redo-ifchange[2]: ERROR: services/dbus-broker.service: Not done.
>> redo-ifchange[2]: ERROR: services/dbus-daemon.service: Not done.
>> redo-ifchange[2]: ERROR: services/system-wide.conf: Not done.
>> redo-ifchange[2]: ERROR: systemd/service-manager.socket: Not done.
>> redo-ifchange[2]: ERROR: convert/per-user/at-spi-dbus-bus.service: Not
>> done.
>> redo-ifchange[2]: ERROR: convert/per-user/gconfd.service: Not done.
>> redo-ifchange[2]: ERROR: convert/per-user/per-user.conf: Not done.
>
> A snapshot of the 1.38 development is in the usual place.  See how far that
> gets you.

Farther, but not to the end:

redo-ifchange[2]: ERROR: services/dbus-broker.service: Not done.
redo-ifchange[2]: INFO: services/dbus-daemon.service: Redone.
redo-ifchange[2]: ERROR: systemd/service-manager.socket: Not done.
redo-ifchange[2]: ERROR: services/system-wide.conf: Not done.
redo-ifchange[2]: INFO: convert/per-user/at-spi-dbus-bus.service: Redone.
redo-ifchange[2]: INFO: convert/per-user/gconfd.service: Redone.
redo-ifchange[2]: INFO: convert/per-user/per-user.conf: Redone.
redo[1]: ERROR: all: Not done.

Oh, look!

$ grep gentoo source/services/*do source/convert/per-user/*do
source/services/dbus-daemon.service.do: gentoo:*)  ext=debian-linux ;;
source/convert/per-user/at-spi-dbus-bus.service.do: gentoo:*)
ext=redhat-linux ;;
source/convert/per-user/gconfd.service.do: gentoo:*)  ext=redhat-linux ;;
source/convert/per-user/per-user.conf.do: gentoo:*)  ext=linux ;;

I could not spend much time looking into that myself once the weekend
was over, but these look like the correct choices. Did someone else
help with that? Well, I don't actually have the gconf and at-spi2-core
packages installed at the moment, but looking at their ebuilds and the
gnome2 eclass, it does look like Gentoo goes with upstream's defaults,
so gconfd-2 and at-spi-bus-launcher would be indeed in /usr/libexec,
like for RHEL / CentOS apparently.

As for the missing .do scripts, if you are going to go this route:

services/dbus-broker.service.do should symlink dbus-broker.service to
dbus-broker.service.debian-linux
systemd/service-manager.socket.do should symlink
service-manager.socket to service-manager.socket.debian-linux
services/system-wide.conf.do should symlink system-wide.conf to
system-wide.conf.linux

G.

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:
> Did someone else help with that?

No.  I am reliant upon you and anyone else who pipes up.

Guillermo:
> if you are going to go this route:

Yes.  I thought about it and it seemed that doing otherwise places the 
burden on the wrong person.  The other approach is to defer running the 
*.do files until the end-user runs the import subsystem in xyr 
$HOME/.config/service-bundles/convert/ directory. That's placing the 
burden of working out how GNOME confd, dbus-daemon, and others are built 
for any given operating system, on every single end-user.  The burden of 
that should be just on the shoulders of the person doing the porting to 
the operating system in question and making the relevant operating 
system packaging.

There's one more to go.  Have a look at the Packages and Ports chapter 
on the copy of the nosh Guide on the WWW site.  (I might move this doco.)

nosh version 1.39

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.39 .

* http://jdebp.eu./Softwares/nosh/

* 
https://www.freebsd.org/news/status/report-2017-07-2017-09.html#The-nosh-Project 


* http://jdebp.info./Softwares/nosh/

I missed announcing 1.38, so this announcement will cover both versions.

These versions see a major addition to the user-space virtual terminal 
subsystem, various other changes in several areas, the completion of 
some items mentioned as placeholders in version 1.37, and some bug fixes.


Completed placeholders
======================

make-read-only-fs is now fully implemented, and is no longer a placeholder.


More service bundles
====================

There are several more additions to the set of service bundles supplied 
with the toolset: connman, ofono, dundee, cntlm, minidlna, powertop, 
alsa-state, alsa-restore, unattended-upgrade-shutdown, apt-daily-update, 
apt-daily-upgrade, LCDd, phpsessionclean, tinysshd, watchman, rngd, 
isnsd, isnsdd, usbmux, and VBoxBalloonCtrl.  atd is now a Linux-only 
service, with the BSDs now having an atrun service.


More packages
=============

The new nosh-run-bcron, nosh-bcron-as-cron-shims, nosh-debian-crontab, 
and nosh-debian-crontab-anacron packages deal in running the services 
and providing the data files for various cron toolsets.  The former two 
deal in bcron, running its services and providing the crontab command as 
an alias for bcrontab; and the latter two (only available for Linux 
operating systems) deal in Debian's /etc/crontab file.

The new nosh-openrc-shims package contains shims for OpenRC's rc-service 
and rc-update commands.  And the new nosh-run-via-open-rc package 
contains OpenRC scripts for running the service manager.

The new nosh-linux-shims package contains shims for commands to be found 
in the non-portable util-linux toolset, such as setterm (more on which 
later).

The Debian desktop and server base -run packages no longer preset ntpd 
and openntpd, on the grounds that a range of such services exist and 
these are not necessarily the installed softwares.


More tools
==========

New commands include getuidgid, userenv-fromenv, setgid-fromenv, envgid, 
printenv, setlogin, console-decode-ecma48, console-control-sequence, 
console-flat-table-viewer, console-input-method, and 
local-stream-socket-connect.

The userenv command is now a combination of two of these new commands, 
getuidgid and userenv-fromenv.  It has also gained options for not 
setting SHELL and USER/LOGNAME.

setlogin sets the login account that is associated with a kernel 
session, as returned by the logname command.

printenv is roughly equivalent to the conventional tool of the same 
name, except that it is a nosh/exec built-in command and that it 
supports several forms of output (including properly quoted rc.conf 
form, NUL-terminated form, and envdir form) in addition to the 
conventional human-readable form.  This built-in command makes a common 
idiom easier.  When combining clearenv, read-conf/envdir, and printenv 
to read a configuration setting, before the advent of the built-in 
command one had to employ `command -v printenv` (because clearenv unsets 
PATH).  Now one can invoke it as simply printenv.

One common use of this idiom is by the toolset's own build system and by 
the external configuration import subsystem, to read things like the 
amalgamated /etc/system-control/convert/rc.conf and an os_version file.  
Further to this, the amalgamated rc.conf now has an os_version setting 
on Linux operating systems, consolidating the code for obtaining that in 
one place.

console-flat-table-viewer is a full-screen TUI viewer for various sorts 
of common flat database tables.  It decodes the vis(3) encoding that is 
employed in various FreeBSD system tables.  It also handles tables that 
use the standard ASCII US, RS, GS, and FS characters.  File separators 
permit a form of continual update and redisplay if used in combination 
with pipes.

local-stream-socket-connect is the AF_LOCAL socket equivalent of 
tcp-socket-connect.


Improvements to existing tools and bug fixes
============================================

The Z shell completions now function better, and now cover a lot more of 
the commands in the toolset.

systemd service unit conversion has been modified to make use of the new 
environment commands.  The conversion tool in particular makes use of 
these when converting per-user Desktop Bus services.  The 
EnvironmentUser extension has been replaced by an EnviromnentUserOnly 
extension, so that User and Group are consistently the sources of the 
user account and primary group. Additional settings now supported by 
convert-systemd-units include RuntimeDirectoryGroup, 
RuntimeDirectoryPreserve, WantsMountsFor, AfterMountsFor, and 
RequiresMountsFor.

convert-systemd-units now also supports %T, %V, and %E expansions and 
snippets files.

By analogy to \S, the login-banner command now also recognizes the \N 
sequence.

The external configuration import subsystem now generates per-user 
service bundles that import user-wide environment variables from a 
${HOME}/config/service-bundles/common pseudo-bundle, allowing users to 
maintain a single environment directory that affects all per-user 
services.  One can thus use rcctl or system-control set-service-env 
against this common pseudo-bundle to set environment variables in all 
per-user services.

Other improvements to external configuration import include the 
automatic generation of service bundles for dbus services, at both the 
system-wide and per-user levels.

The N and P actions of console-multiplexor-control when applied to 
kernel virtual terminals now work properly.

The external configuration input subsystem now imports defaultrouter 
properly from the amalgamated rc.conf.

The cleanX service no longer generates incorrect symbolic links in /tmp.

ttylogin services on user-space virtual terminals were sometimes 
(depending from exactly what order things ran in) causing a loop because 
of vhangup().  This has been fixed.

And a spelling error in the names of the UNIXREMOTEEUID and 
UNIXREMOTEEGID UCSPI-UNIX environment variables has been fixed.


Input methods
=============

* http://jdebp.eu./Softwares/nosh/japanese-input-methods.html

Providing a "front-end processor" for running "input methods" has been a 
to-do item on the roadmap for a long time.  It has always been a goal to 
provide more than just U.S.-centric mechanisms in user-space virtual 
terminals, and this already ranges from allowing one to use fonts with 
large Unicode glyph repertoires through to providing the full ISO 9995-3 
common secondary keyboard group.  The problems with input method support 
were that all of the existing systems that I could tie into required X11 
servers, X11 libraries, or direct low-level access to the framebuffer.

Then I discovered OpenVanilla.

The .cin file mechanism is table-based, does not involve plug-ins 
needing direct drawing access to the framebuffer, and is common across 
that and at least 8 different other tools.  It is now common to 
user-space virtual terminals, too.  Just as one can take a SCO Unix or 
FreeBSD kbdmap file and (after conversion) use it with user-space 
virtual terminals, one can take other people's existing .cin files and 
use them.

Other people have written quite a lot of them, moreover.  The blurb page 
on Japanese input methods hyperlinked earlier focuses on Japanese, but 
there is a wide range of .cin files available from plenty of sources, 
from Hangeul Jamo through Array40 to Esperanto.

A new console-input-method command is at the heart of new input method 
services, which plumb in to user-space virtual terminals in between the 
terminal emulator(s) and the realizer(s).  The pre-supplied "head0" 
user-space virtual terminal now has an input method service plumbed in.  
The nosh Guide documents input method front-end processors.  And a new 
console-input-method-control command can be used to control them in a 
few basic ways.

In line with the philosophy of adopting and adapting existing ideas, 
visible from the aforementioned keyboard map and .cin files as well as 
the whole building upon the architecture of daemontools notion, the 
input method front-end processor provides various control key chords and 
function keys that are roughly compatible with other systems, such as 
Microsoft's Japanese IME in Windows and the OSF/1 IMLIB.  One can, of 
course, drive it with the extra keys that are dedicated to the purpose 
on a JIS 106/109-key keyboard or a Korean 103/106-key keyboard.  There 
should be not too much change required to existing typing habits.


Other terminal improvements
===========================

In addition to the aforementioned documentation of front-end processors, 
the Guide also now documents how "head0" works (although this is just 
one way in which one can set up virtual terminals) and has a new chapter 
on some of the choices of fonts, keyboard maps, and input methods that 
are available from various sources for user-space virtual terminals.

(In a separate project, I have been working on fixes to FreeBSD's 
vtfontcvt that permit it to convert Ubuntu Monospace, which as the Guide 
documents it currently cannot handle.)

* http://jdebp.eu./Proposals/linux-kvt-manual-pages.html

There are new manual entries for TERM(7), linux-vt(4), linux-console(4), 
and TerminalCapabilities(3).

No utility other than console-ncurses-realizer uses NCurses and terminfo 
any more.  They have flaws in how they model real terminals and terminal 
emulators.  All other TUI tools are built around a different library, 
that decodes terminal input using a proper full UTF-8 ECMA-48 state 
machine rather than by limited and faulty pattern matching, and that is 
geared towards primarily ECMA-48 and ITU T.416 output (with various 
common DEC private extensions).  This enables better handling of cursor, 
editing, and calculator keypad keys with modifiers; full recognition of 
"application" and "normal" modes on the cursor and calculator keypads; 
full use of 24-bit colour where the terminal supports it; and use of the 
DEC Locator or XTerm mouse.

One such is a new realizer, console-termios-realizer, another 
realize-a-terminal-on-a-terminal realizer that is more capable than 
console-ncurses-realizer.

console-fb-realizer now has better fallback behaviour in the event of 
not being supplied a keyboard map, which has been a fairly common 
configuration error.  It now falls back to the U.S. English 
International keyboard map, rather than to an empty one where no keys do 
anything.

Other improvements lie in extended keys, on the cursor and calculator 
keypads.  console-terminal-emulator now fully supports individually 
switching these two keypads between "application" and "normal" modes, 
and fully supports separate control sequences sent by keys for each 
mode.  To align with this, the "fkey49" to "fkey61" actions in SCO 
Unix/FreeBSD keyboard maps have changed in console-convert-kbdmap to 
distinguish the cursor keypad from the calculator keypad, and 
console-fb-realizer collaborates in ensuring that the terminal 
emulator(s) receive as much modifier state information accompanying such 
keys as is appropriate.

The keyboard map files themselves now follow a new naming convention.  
The external configuration import subsystem will not delete any old 
keyboard map files that you might still be using in a running realizer 
service, so it is left to you to clean them up. The new naming 
convention allows for, say, different "Japanese" maps to be used with a 
101/104-key keyboard and a 106/109-key keyboard.

The user-space virtual terminal subsystem now also has the notion of 
accelerator key input messages, i.e. alphanumeric key input with the ALT 
key modifier.  These are recognized by the realizers and the terminal 
emulator generates escape sequences for them.  Note that these escape 
sequences, which are neither ECMA-48 nor DEC VT/SCO standard, conflict 
with ECMA-48 7-bit control aliases.

console-decode-ecma48 is a debugging aid for text sent to/from 
terminals.  It decodes a sequence of ECMA-48 characters, control 
characters, escape sequences, and control sequences into a 
human-readable representation.  It can handle both ECMA-48 output (as 
sent to terminals) and ECMA-48 input (as received from terminals), and 
recognizes as an extension various common additional control sequences, 
such as the control sequences for function keys generated by the SCO 
Unix console, and a bunch of DEC VT private control sequences.

* https://unix.stackexchange.com/a/491883/5132

console-control-sequence provides a human-readable way of emitting 
common ECMA-48, DEC VT, AIXterm, and XTerm control sequences.  It is 
aliased as setterm and is to a large extent a workalike for the setterm 
from the util-linux package, except that it is portable where util-linux 
is not.  It is both portable to other operating systems, and portable to 
other terminals.  It does not implement the few things that util-linux 
setterm does that are specific to Linux and to Linux's built-in teminal 
emulator, but conversely it implements quite a number of standard 
ECMA-48 and DEC VT control sequences that the util-linux setterm does 
not.  These include turning the calculator keypad to/from application 
mode; switching to/from the XTerm alternate screen buffer; turning DEC 
Locator reports on/off; turning XTerm mouse reports on/off; setting the 
cursor shape; changing the mappings of the Delete and Backspace keys; 
using the strikethrough, italic, and invisible attributes; turning 
background colour erase on/off; setting the underline type; indirect 
8-bit colour; direct 24-bit colour; and DEC VT soft reset.

vc-reset-tty and console-resize are now implemented by invoking 
console-control-sequence, removing the hardwiring of control sequences 
from both.

console-terminal-emulator now recognizes a few additional control 
sequences, such as DECST8C (used by console-control-sequence to set 
regular tabstops if the interval length is 8).  It also has tabstops at 
8 column intervals in its reset state.  Both the FreeBSD and Linux 
kernel virtual terminals do this, even though real DEC VTs do not, so 
user-space virtual terminals do too.

The --keep-term option to clearenv now retains a few more 
terminal-related environment variables.


===========
This way up

And finally, a feature that you will not want.

* https://unix.stackexchange.com/a/465166/5132

Over many years, people have regularly asked for a terminal emulator 
where the line progression goes from bottom to top.  The idea that they 
have is that they will find this easier than top to bottom.  In response 
to a recent question in the same vein on Stack Exchange, I have given 
console-fb-realizer, console-ncurses-realizer, and 
console-termio-realizer the ability to realize displays with the line 
progression reversed.

Having used this, it seems to me that people only want this because they 
have not tried it.  Trying it makes a convincing case for not using it.  
So I have retained this feature in order that, at long last, there is an 
actual terminal system with this feature that people can try, and come 
to the same realization.  (-:

nosh version 1.40

[Jonathan de Boyne Pollard]

[-- Attachment #1: Type: text/plain, Size: 4728 bytes --]

The nosh package is now up to version 1.40 .

  *

    http://jdebp.eu./Softwares/nosh/

  *

    http://jdebp.info./Softwares/nosh/

  *

    https://github.com/freebsd/freebsd-quarterly/blob/master/2018q4/nosh.md

This version sees changes to the doco, improvements to network 
configuration, and a change to machine ID generation.


FreeBSD binaries

================

I plan for this to be the last release with binaries built on FreeBSD 
10.  I am going to upgrade the build machine.


|ifconfig|

========

  *

    https://unix.stackexchange.com/a/504084/5132

There is now an |ifconfig| command in the toolset, with a command-line 
interface and output similar to the FreeBSD |ifconfig|. It is primarily 
intended for use on non-FreeBSD systems, to provide a FreeBSD-like 
|ifconfig| where one does not have the actual FreeBSD tool.  The 
|ifconfig@*| services generated by the external configuration import 
subsystem make use of it on Linux operating systems, allowing the 
services themselves to be pretty much the same across platforms.

It handles multiple (unlabelled) addresses per interface and both IP 
version 6 and IP version 4 addressing, which are two of the 
long-standing complaints against the old |ifconfig| programs from GNU 
inetutils <https://gnu.org/software/inetutils/> and NET-3 net-tools 
<https://sourceforge.net/projects/net-tools/>.  It has no notion of 
overwriting a single "primary" address.  It has a |broadcast1| flag for 
calculating the broadcast address from the prefix length and address.  
It prefers the new (since 1993) notation for IP version 4 network 
masks.  It can do the FreeBSD style of EUI-64 address assignment for IP 
version 6 with an |eui64| flag.

And it colours its output if writing to a terminal.  (-:


Machine ID generation

=====================

FreeBSD from 2007 onwards used the SMBIOS system UUID from the machine 
firmware as a fallback source for a machine ID. |setup-machine-id| prior 
to this release of the toolset would do the same for compatibility.  
This has now been removed from |setup-machine-id|. The privacy problems 
that it entails have turned out to outweigh what little utility it had.

Systems that would have fallen back upon the SMBIOS system UUID will now 
fall back to creating UUIDs using the C library.  Note that the FreeBSD 
C library still uses MAC addresses to create UUIDs.  The OpenBSD and GNU 
C libraries use CSPRNGs.

There is also now an |erase-machine-id| command that resets all of the 
machine ID storage locations set by |setup-machine-id| to a nil UUID.  
The |machine-id| service now calls |erase-machine-id| at shutdown.

Thus: Machine IDs (when using the supplied service bundles) now have a 
lifetime from bootstrap to shutdown, will not persist across reboots, do 
not reveal the SMBIOS system UUID and are not constant and correlatable 
because of it even when explicitly wiped, and can still reveal MAC 
addresses on FreeBSD.

The new |machine-id|(7) manual page lists some of the known users of 
machine IDs, explains where machine IDs are stored, and gives some of 
the history of machine IDs.


Square mode

===========

Square mode is now switchable in |console-terminal-emulator|, using DEC 
Private Mode 1369. |console-control-sequence| has a |--square| option 
for changing it.


Other tools

===========

|ucspi-socket-rules-check| has gained the ability to check |uid/self/| 
and |gid/self/| subdirectories when handling UCSPI-UNIX connections.


Doco

====

The Guide now includes the original command manuals, written in DocBook 
XML.  These are directly readable using a GUI WWW browser and the 
supplied stylesheet.  The conversions to HTML are still supplied, but 
reading the original DocBook XML format is better.

TUI WWW browsers such as lynx cannot read DocBook XML.  Their deficiency 
has inspired a new |console-docbook-xml-viewer| tool that parses and 
displays the manual pages with a simple full-screen interface on a 
terminal.  This can of course display other DocBook XML manual pages as 
well.


External configuration import improvements

==========================================

The external configuration import subsystem now allows various 
extensions in a |/etc/network/interfaces| file, including |ipv4ll| 
stanzas (which will cause |avahi-autoipd| services to be set up) and 
|eui64| stanzas.  A "|broadcast +|" setting is now transformed into the 
aforementioned |broadcast1| flag for |ifconfig|.

It also now once more treats |false| for the login shell as signifying a 
non-personal user account.


More service bundles

====================

There are a few more service bundles in this release, including ones for 
Laurent Bercot's mdev, for two-ftp, and for NcFTPd.