s6-linux-init: Actions after unmounting filesystems

s6-linux-init: Actions after unmounting filesystems

[Guillermo]

For the 0.4.x.x series of s6-linux-init, the shutdown sequence model
was that s6-svscan replaced itself with a 'stage3 init' that ran as
process 1 until the machine halted, powered off, or rebooted. The
stage3 init was an execline script.

s6-linux-init-1.0.0.0 and later, among other things, has changed the
shutdown sequence model. The new one is that a supervised 'shutdown
daemon', s6-linux-init-shutdownd, performs all shutdown tasks, and
s6-svscan remains process 1 until the machine halts, powers off, or
reboots. s6-linux-init-shutdownd is a C program.

In both models, all processes are killed with a 'kill(-1, SIGTERM)'
call and then a 'kill(-1, SIGKILL)' call, and then filesystems are
unmounted. For the 0.4.x.x series, this is done by having the stage3
init run a program. Because the stage3 init runs as process 1, it
survives the SIGKILL. For s6-linux-init-1.0.0.0 and later, this is
done directly by s6-linux-init-shutdownd. Because
s6-linux-init-shutdownd is part of a supervision tree rooted in
process 1, it is respawned after the SIGKILL, and then completes the
shutdown sequence.

There are certain setups that require doing something after
filesystems are unmounted. Two examples are LVM logical volumes and
LUKS encrypted volumes, but I suppose there must be more cases. In any
such setup, the shutdown sequence would include a 'vgchange -a n'
command or 'cryptsetup close' command after filesystems are unmounted.
Runit-based Void Linux does this in /etc/runit/3:

* https://github.com/void-linux/void-runit/blob/0566391df8c9c93f75ad99d94c8a19abe379908b/3#L37

deactivate_vgs and deactivate_crypt are defined in /etc/runit/functions:

* https://github.com/void-linux/void-runit/blob/0566391df8c9c93f75ad99d94c8a19abe379908b/functions#L38

Gentoo, and I suppose that other OpenRC-based distributions as well,
leave this to the service manager. OpenRC-provided 'localmount'
service uses 'fuser -m -k' and umount commands in its stop() function
to unmount filesystems, and 'killprocs' service takes care of every
remaining process using OpenRC's internal kill_all command. Gentoo
then adds services named 'lvm' and 'dmcrypt' to OpenRC's boot
runlevel, and service ordering at shutdown, enforced by specifications
in service's depend() functions and OpenRC's runlevels model, is
localmount < root (a no-op) < fsck (a no-op generally) < {lvm,
dmcrypt} < killprocs < mount-ro, where "<" means "precedes" here.

How would something like this fit in s6-linux-init's new shutdown
model? In the old one, just like in runit's case, I suppose it could
be done in 'shutdown hooks' called by the execline stage3 init.
Something like:

foreground {
   redirfd -r 0 shutdown-hooks.conf
   forstdin shutdown_hook
   importas -u shutdown_hook shutdown_hook
   if -nX { $shutdown_hook }
      echo stage3 init: warning: shutdown hook \"${shutdown_hook}\" failed
}

I also looked at Adélie Linux, which currently uses
s6-linux-init-1.0.2.1 + OpenRC if it is configured to do so, and it
seems to copy Gentoo in this case. It also has lvm and dmcrypt OpenRC
service scripts:

* https://code.foxkit.us/adelie/packages/raw/b1eb50599e4db7eb4501af75cbbfa22007081ea5/system/lvm2/lvm.initd
* https://code.foxkit.us/adelie/packages/raw/49602ddb82f87067bbb945fc2a814a8eb6d6aabe/system/cryptsetup/dmcrypt.initd

I can't verify without actually installing Adélie, but I suppose these
are also boot runlevel services. However, it appears that Adélie's
/etc/s6-linux-init/skel/rc.shutdown just asks OpenRC to enter a custom
runlevel named 'empty'. Because this isn't the shutdown runlevel, if I
understand correctly this would neither stop boot runlevel services,
nor start shutdown runlevel services, so OpenRC's localmount,
killprocs and mount-ro would not interfere with s6-linux-init's
shutdown procedure. But doesn't this also mean that the vgchange and
cryptsetup invocations in lvm's and dmcrypt's stop() functions would
never happen?

G.

Re: s6-linux-init: Actions after unmounting filesystems

[Laurent Bercot]

>There are certain setups that require doing something after
>filesystems are unmounted. Two examples are LVM logical volumes and
>LUKS encrypted volumes, but I suppose there must be more cases. In any
>such setup, the shutdown sequence would include a 'vgchange -a n'
>command or 'cryptsetup close' command after filesystems are unmounted.

Ah, of course it had to be more complex... Sigh.

There are two ways to proceed here:

- If a filesystem can track all the processes that have a handle on
it, it is possible to have it be mounted/unmounted symmetrically by
the service manager. At unmount time, kill the processes that would
block the unmount operation, then perform the unmount, then run the
additional commands. In that case it's all done at the service manager
level, s6-linux-init doesn't have to do anything.

- There could be a hook in the autogenerated stage 4 script, which
runs a user-provider script, something like rc.shutdown.after-umount.
I don't much like giving control to a user script at that level, when
there are no services running and no mounted filesystems, possibly
not even /proc or /sys, and when a hang in a user script could
very well prevent a clean reboot; but it's the best I can do.
Admins/distros would have to make sure the deactivate_* functions
only call binaries that are on the root filesystem.

Thoughts?

The asymmetry of mounting and unmounting filesystems is really a pain
in the ass for the design of an init/shutdown sequence. I wanted to
keep the shutdown as quick, minimal, and reliable as possible, but it
seems there's no way to do so with those snowflake filesystems. :/

--
Laurent

Re: s6-linux-init: Actions after unmounting filesystems

[Casper Ti. Vector]

On Sat, Aug 17, 2019 at 11:01:35PM +0000, Laurent Bercot wrote:
> The asymmetry of mounting and unmounting filesystems is really a pain
> in the ass for the design of an init/shutdown sequence. I wanted to
> keep the shutdown as quick, minimal, and reliable as possible, but it
> seems there's no way to do so with those snowflake filesystems. :/

Sorry if I read this thread too hastily, but why not just keep /proc etc
mounted, as was seemingly the way with s6-linux-init <=v0.4.x.x (and
therefore slew)?  Since the asymmetry is by nature, simply respecting it
appears to be one minimalist way.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: s6-linux-init: Actions after unmounting filesystems

[Laurent Bercot]

>Sorry if I read this thread too hastily, but why not just keep /proc etc
>mounted, as was seemingly the way with s6-linux-init <=v0.4.x.x (and
>therefore slew)?  Since the asymmetry is by nature, simply respecting it
>appears to be one minimalist way.

The umount command basically performs a "umount -a". I can add a list
of filesystems that should be kept, but it's more ad-hoc code. It's
ugly.
But I guess the whole situation is ad-hoc anyway, so elegance isn't
exactly what we're going for here :/

--
Laurent

Re: s6-linux-init: Actions after unmounting filesystems

[Casper Ti. Vector]

On Sun, Aug 18, 2019 at 06:26:05AM +0000, Laurent Bercot wrote:
> The umount command basically performs a "umount -a". I can add a list
> of filesystems that should be kept, but it's more ad-hoc code. It's
> ugly.

I see; slew always does it this way:
<https://gitlab.com/CasperVector/slew/blob/master/init/umount.rc>.
[BTW, I really dislike the slow (because bloated?) frontend of GitLab.]
And since it is just a script, the user certainly has the final say.

-- 
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C

Re: s6-linux-init: Actions after unmounting filesystems

[Samuel Holland]

On 8/18/19 1:26 AM, Laurent Bercot wrote:
>> Sorry if I read this thread too hastily, but why not just keep /proc etc
>> mounted, as was seemingly the way with s6-linux-init <=v0.4.x.x (and
>> therefore slew)?  Since the asymmetry is by nature, simply respecting it
>> appears to be one minimalist way.
> 
> The umount command basically performs a "umount -a". I can add a list
> of filesystems that should be kept, but it's more ad-hoc code. It's
> ugly.
> But I guess the whole situation is ad-hoc anyway, so elegance isn't
> exactly what we're going for here :/

Simply excluding filesystems doesn't help when the root filesystem is on one of
these devices that needs teardown actions after being unmounted. In that case,
the only workable solution is to have PID1 pivot_root() to a tmpfs with the
teardown/reboot tools in it. That way you can actually fully unmount the former
root filesystem.

Under the current architecture, I think it is ideal to maintain the
mount/unmount symmetry as much as possible. This is done in your service manager
via dependencies. Have a oneshot for each filesystem that needs to be mounted,
and (when applicable) have that oneshot depend on another oneshot that
creates/destroys the underlying lvm/md/dm-crypt device. Then the dm device will
be created before mount, and destroyed after unmount.

Yes, unmounting will fail if the filesystem is in use (or already unmounted),
and destroying will fail if the filesystem is still mounted, but that's okay
because a `down` script should be idempotent and /always/ return success. Even
when there are failures, this will minimize the number of loose ends left when
shutdownd takes over. The unmounting done by shutdownd should be treated as a
last resort.

As a side note: what if there was a oneshot that did `kill -1` when brought
down, and this oneshot depended on all of your filesystem mounts? Other than the
obvious problem of s6-rc-update nuking your system, would it be possible to make
s6-rc recover from being nuked and continue a shutdown?

--Samuel

Re: s6-linux-init: Actions after unmounting filesystems

[Laurent Bercot]

>Simply excluding filesystems doesn't help when the root filesystem is on one of
>these devices that needs teardown actions after being unmounted. In that case,
>the only workable solution is to have PID1 pivot_root() to a tmpfs with the
>teardown/reboot tools in it. That way you can actually fully unmount the former
>root filesystem.

Are there systems in the real world that actually work like that? That
need pivoting into a "shutdownramfs" in order to be able to unmount the
rootfs and perform teardown operations on it? This is doable, of course,
but sounds too complex and too brittle. You'd need more than a fsck to
recover after a power loss, for instance.

This is a real question, because if s6-l-i has to address that case,
it needs some deeper thought about stage 4 management - it essentially
means that the after-shutdown hook must be exec'ed, not called, because
paths in the stage 4 script may not be valid anymore after a pivot_root.
(There's just a call to s6-l-i-hpr -f, but it kinda is a critical part
of the shutdown procedure and it has to succeed.)


>Under the current architecture, I think it is ideal to maintain the
>mount/unmount symmetry as much as possible. This is done in your service manager
>via dependencies. Have a oneshot for each filesystem that needs to be mounted,
>and (when applicable) have that oneshot depend on another oneshot that
>creates/destroys the underlying lvm/md/dm-crypt device. Then the dm device will
>be created before mount, and destroyed after unmount.
>
>Yes, unmounting will fail if the filesystem is in use (or already unmounted),
>and destroying will fail if the filesystem is still mounted, but that's okay
>because a `down` script should be idempotent and /always/ return success. Even
>when there are failures, this will minimize the number of loose ends left when
>shutdownd takes over. The unmounting done by shutdownd should be treated as a
>last resort.

I agree that keeping as much symmetry as possible is cleaner, but
unmounts failing because the fs is still in use is not an exception,
it's _the common case_ if no nuke is performed before the unmount. And
if a "destroy" action has to be performed after an unmount, and cannot
be done if the unmount fails, then it still needs to be performed after
the stage 4 unmount - so it still needs to be registered in the hook,
there's no way around it.


>As a side note: what if there was a oneshot that did `kill -1` when brought
>down, and this oneshot depended on all of your filesystem mounts? Other than the
>obvious problem of s6-rc-update nuking your system, would it be possible to make
>s6-rc recover from being nuked and continue a shutdown?

  s6-rc-update would only nuke your system if there was a change below
the services that mount your filesystems :)

  That said, I don't like the idea of sending a nuke while the service
manager is still active. There would be a way to make s6-rc recover:
have a special service that checks a file when it starts, and if the
file is there, it's a recovery mark and an instruction to run the rest
of the procedure. (It's exactly what s6-l-i-shutdownd is doing to run
stage 4 after it gets killed by the stage 3 nuke.) But it would mean
an additional ad-hoc service, and as cool as it is to see a
supervision tree automatically restart processes, I'm not a fan of
triggering the Apocalypse while we're still in the process of tidying
up the world in an orderly fashion. Let's tidy up what we can and
exit the stage before killing everyone. That's just being civilized.

  I'm afraid the best utility/complexity ratio is just to call a hook
in stage 4, and declare it the responsibility of the hook writer to
ensure the hook doesn't hang - it has to either exit in a reasonable
time or perform a hard reboot itself (e.g. in a pivot_root situation).

  Now the fun part for me is to find a way for s6-l-i-umountall to
leave the proper filesystems mounted. It's not as straightforward as
it seems: if /dev is a symlink to, say, /mnt/tmpfs/dev, then you want
to keep /mnt/tmpfs/dev, even if it means you have to keep /mnt/tmpfs.
But if you have a second dev mount on /usr/local/mychroot/dev, then
you want to unmount that one, so you can unmount /usr/local. I suppose
I can count the number of devtmpfs, procfs and sysfs, and only keep
one of each (the first that was mounted), but I have to think more
about it to make sure it catches everything.

--
  Laurent

Re: s6-linux-init: Actions after unmounting filesystems

[Guillermo]

El dom., 18 ago. 2019 a las 15:36, Laurent Bercot escribió:
>
> >Simply excluding filesystems doesn't help when the root filesystem is on one of
> >these devices that needs teardown actions after being unmounted. In that case,
> >the only workable solution is to have PID1 pivot_root() to a tmpfs with the
> >teardown/reboot tools in it. That way you can actually fully unmount the former
> >root filesystem.
>
> Are there systems in the real world that actually work like that? That
> need pivoting into a "shutdownramfs" in order to be able to unmount the
> rootfs and perform teardown operations on it? This is doable, of course,
> but sounds too complex and too brittle. You'd need more than a fsck to
> recover after a power loss, for instance.

I know that there are people that have the rootfs on an LVM logical
volume or a LUKS encrypted volume, yes. However, those are specialized
setups. I don't use one myself, I think they are risky, and I don't
know the details, but I think they are handled with a (mandatory)
initramfs. Perhaps someone else knows this better. Our friends from
the systemd project have thought of that as well (the 'shutdownramfs'
would be the initramfs that is kept mounted at /run/initramfs):

* https://www.freedesktop.org/wiki/Software/systemd/InitrdInterface

But I think of this as a separate problem, that maybe also means
revisiting the decision of not having s6-svscan do its finish
procedure. I didn't want to mention all the complicated cases at once
:)

>   Now the fun part for me is to find a way for s6-l-i-umountall to
> leave the proper filesystems mounted. It's not as straightforward as
> it seems: if /dev is a symlink to, say, /mnt/tmpfs/dev, then you want
> to keep /mnt/tmpfs/dev, even if it means you have to keep /mnt/tmpfs.
> But if you have a second dev mount on /usr/local/mychroot/dev, then
> you want to unmount that one, so you can unmount /usr/local. I suppose
> I can count the number of devtmpfs, procfs and sysfs, and only keep
> one of each (the first that was mounted), but I have to think more
> about it to make sure it catches everything.

Why not just leaving all of them alone? s6-linux-init-umountall
already warns about, but then just ignores, umount(2) failures. What
happens if you try to unmount /mnt/tmpfs, and you have a devtmpfs
mounted on /mnt/tmpfs/dev that you have previously skipped?

G.

Re: s6-linux-init: Actions after unmounting filesystems

[Oliver Schad]

[-- Attachment #1: Type: text/plain, Size: 931 bytes --]

On Sun, 18 Aug 2019 16:28:40 -0300
Guillermo <gdiazhartusch@gmail.com> wrote:

> I know that there are people that have the rootfs on an LVM logical
> volume or a LUKS encrypted volume, yes. However, those are specialized
> setups.

In some distributions it's a default AFAIR. I know a lot of people who
do that on workstations and servers.

More specialized are non-LUKS crypto setups which works with an
external USB stick which is removed after start for plausible denial of
knowledge of a crypto setup.

I know some crypto setups which broke through systemd.

BTW: a good sign to do it another way is that our systemd friends are
thinking about it. ;-)

Best Regards
Oli

-- 
Automatic-Server AG •••••
Oliver Schad
Geschäftsführer
Turnerstrasse 2
9000 St. Gallen | Schweiz

www.automatic-server.com | oliver.schad@automatic-server.com
Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: s6-linux-init: Actions after unmounting filesystems

[Guillermo]

El sáb., 17 ago. 2019 a las 20:01, Laurent Bercot escribió:
>
> - If a filesystem can track all the processes that have a handle on
> it, it is possible to have it be mounted/unmounted symmetrically by
> the service manager.

I don't think there are filesystems that can do that?

> At unmount time, kill the processes that would
> block the unmount operation, then perform the unmount, then run the
> additional commands. In that case it's all done at the service manager
> level, s6-linux-init doesn't have to do anything.

That's OpenRC's approach, except it does not rely on filesystem
features. It just uses 'fuser -m -k'. But it leads to code that's
quite ugly I think, compared to the simplicity of s6-linux-init's
'kill(-1, SIGTERM)' + 'kill(-1, SIGKILL)':

* https://github.com/OpenRC/openrc/blob/882c6bf3bcaba6903d9dc593f8ae41e505b4e4e7/sh/rc-mount.sh

(mountinfo is another OpenRC internal command)

> - There could be a hook in the autogenerated stage 4 script, which
> runs a user-provider script, something like rc.shutdown.after-umount.

This is probably better, but I'd also like to hear other opinions.

> I don't much like giving control to a user script at that level, when
> there are no services running and no mounted filesystems, possibly
> not even /proc or /sys,

You'd probably have to make some exclusions in
s6-linux-init-umountall, like Casper said. Maybe parse the fstype
field in /proc/mounts lines and omit sysfs, proc, tmpfs and devtmpfs?
BTW, OpenRC also does exclusions, via --skip-point-regex and
--skip-fstype-regex options passed to mountinfo (see do_unmount
invocations in services localmount and mount-ro). I admit this
complicates s6-linux-init-umountall.

> and when a hang in a user script could
> very well prevent a clean reboot

I haven't thougt of that. Ouch. Maybe a mechanism like the one
s6-supervise uses to limit execution of the finish file in
s6-linux-init-shutdownd, or some shutdown hook runner? I admit it
complicates s6-linux-init-shutdownd :)

> Admins/distros would have to make sure the deactivate_* functions
> only call binaries that are on the root filesystem.

It is obvious to me that it must be that way, but in this era in which
some distributions want to put every executable in /usr/bin and mount
/usr from an initramfs, who knows :P But that's the distribution's
job, I agree.

G.


G.

Re: s6-linux-init: Actions after unmounting filesystems

[Brett Neumeier]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

On Sat, Aug 17, 2019 at 2:20 PM Guillermo <gdiazhartusch@gmail.com> wrote:

> There are certain setups that require doing something after
> filesystems are unmounted. Two examples are LVM logical volumes and
> LUKS encrypted volumes, but I suppose there must be more cases. In any
> such setup, the shutdown sequence would include a 'vgchange -a n'
> command or 'cryptsetup close' command after filesystems are unmounted.
>

How important are these cases?

I have not inspected the code to see what cryptsetup close does, aside from
remove the encrypted device mapping for the encrypted volume; nor have I
looked at the LVM code to see if it does anything other than remove device
mapping. But in either case, what I generally do on my systems is simply
unmount volumes (or in the case of the root filesystem, remount it
read-only), sync to make sure all dirty buffers are written, and then just
shut down/reboot/whatever.

Leaving LVM and encrypted volumes active has never caused any problems for
me.

It seems worthwhile to ask whether those examples are real problems that
need to be addressed, since all of the proposed solutions have some level
of hairiness about them. And if those examples are *not* real issues, it
might be worthwhile to ask if there are other examples that *are*.

-- 
Brett Neumeier (bneumeier@gmail.com)