Scripting Stage 3 and 4

Scripting Stage 3 and 4

[Paul Sopka]

[-- Attachment #1.1.1: Type: text/plain, Size: 2451 bytes --]

Hey everybody,

I scripted shutdown stage 3 and 4 based on s6-linux-init-shutdownd and 
it works quite well,
even though being VERY rough around the edges currently:
(Linux-powertool is a minimal program that just makes the syscall to 
halt, poweroff or reboot)

| #!/bin/execlineb -P
|
| multisubstitute {
|     importas -SD /etc/s6-rc G6CONFIGDIR
|     importas -SD /run G6LIVEDIR
| }
|
| fdmove -c 2 1
| redirfd -w 1 /dev/console
| redirfd -r 0 /dev/null
|
| tryexec { ./stage3b }
|
| foreground { rm -f fifo }
| foreground { mkfifo -m 0700 fifo }
| foreground { fdmove 1 3 echo "" }
| redirfd -r 0 fifo
| withstdinas -E DIRECTIVE
|
| if { eltest \\\\${DIRECTIVE} =~ ^-[hpr]$ }
|
| foreground { s6-rc -l ${G6LIVEDIR}/s6-rc -v2 -bDa -t 10000 change }
| foreground {
|     pipeline { ls -1 ${G6LIVEDIR}/service/ }
|     pipeline { sed /g6-shutdown/d }
|     forstdin -pE SERVICE
|         mv -f ${G6LIVEDIR}/service/${SERVICE} 
${G6LIVEDIR}/service/.${SERVICE}
| }
| foreground { s6-svscanctl -h ${G6LIVEDIR}/service }
|
| foreground {
|     redirfd -w 1 stage3b
|     echo "#!/bin/execlineb
|         tryexec { ./stage4 }
|         foreground { mv stage4p stage4 }
|         foreground { kill -SIGCONT -- -1 }
|         foreground { sleep 1 }
|         foreground { echo sending all processes the KILL signal... }
|         foreground { kill -SIGKILL -- -1 }"
| }
| foreground { chmod o+x stage3b }
|
| foreground {
|     redirfd -w 1 stage4p
|     echo "#!/bin/execlineb
|         foreground { mount -o remount,ro / }
|         foreground { umount -a }
|         linux-powertool ${DIRECTIVE}"
| }
| foreground { chmod o+x stage4p }
|
| foreground { echo "sending all processes the TERM signal..." }
| foreground { kill -SIGTERM -- -1 }

Now I was wondering why you, Laurent recommend against using the finish 
script of s6-svscan
for the shutdown procedure (here: 
https://skarnet.org/software/s6/s6-svscan-1.html#stage3).

Using that would make good portion of my implementation more elegant and 
simple,
especially all the dance around creating scripts to be ran by "tryexec" 
after being restarted upon being signaled to stop...

I figured that other people might also have some interest in this,
so I am posting the question here.

Regards,

Paul


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3195 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Scripting Stage 3 and 4

[Laurent Bercot]

>Now I was wondering why you, Laurent recommend against using the finish script of s6-svscan
>for the shutdown procedure (here: https://skarnet.org/software/s6/s6-svscan-1.html#stage3).
>
>Using that would make good portion of my implementation more elegant and simple

  No, it would not, and the link you quoted explains why.

  As long as you have things to do, and especially if you have several
processes involved - as you have in an execline script - you want to
keep a real init system as pid 1, not a shell or a random sequence of
executables as would happen with execline. You don't want to leave
zombies around potentially preventing unmounts, you want to be able
to recover and get a shell until the very end, and you don't want your
pid 1 to accidentally get killed. If you leave s6-svscan running, you
have all this for free; if you terminate it and run your shutdown
sequence as its finish script, you need to pay attention to this for
the entirety of stages 3 and 4.

  I tried both approaches. Keeping s6-svscan as pid 1 until the very
end was by far the safer, less brittle option. Even if it means you
need to add a little dance to have the supervision tree restart what
you want when you want it and nothing else - the peace of mind it
brings is priceless.

--
  Laurent

Re: Scripting Stage 3 and 4

[Paul Sopka]

[-- Attachment #1.1.1: Type: text/plain, Size: 1641 bytes --]

Thanks for the answer, I understand it better now.
To be honest I had to get a little bit used to this unconventional way 
of shutting down.

I found these arguments a little weak:

> you want to be able
> to recover and get a shell until the very end, and you don't want your
> pid 1 to accidentally get killed
Since you can get a shell easily in the finish script too:

| if -n { mount -o remount,ro / }
| sh

And accidentally killing PID 1 would break the supervision tree based 
approach too, right?

But getting a more advanced recovery method up,
e.g. an ssh server when you only have remote access
or an agetty instead of PID 1 sh on a desktop machine,
will be more reliable under a supervision tree, I can see that.

Further, I was wondering about how to implement such a fallback, would:

| if -n { mount -o remount,ro / }
| foreground { s6-svc -U /run/service/recovery }

(With what "recovery" is being a decision of the sysadmin)
Something like this hold up well enough?

One could also argue that the finish script of s6-svscan is really a 
last resort and,
if something can be implemented well in another way (as is the case here),
it should be.

Anyhow, you have convinced me that keeping the supervision tree is the 
better way.

Finally, if there is anything that could fundamentally be done better in 
the script,
please tell me. This is to everybody here, I am happy about every 
GNU'ism or util-linux'ism
pointed out in any of the scripts I post,
as I try to make all my scripts work with pure POSIX + ( ubase | toybox 
| busybox ).

I wish you all a nice weekend,

Paul


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3195 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re[2]: Scripting Stage 3 and 4

[Laurent Bercot]

>Since you can get a shell easily in the finish script too:
>
>| if -n { mount -o remount,ro / }
>| sh

  With no tty? and not even a redirection from/to /dev/console? That's
going to be a tough recovery ;)
  And sure, you could wrap all your commands in a big if block. But my
point is that these are *things you need to think about*. I'd rather
not have to think about new things at shutdown time when I don't have
to. Why change something and duplicate logic when you could just do
nothing and keep what's already working and has worked for the whole
lifetime of the machine?

  The key to understanding how to manage a shutdown, I think, is to
realize that boot and shutdown *are not symmetrical*, and so, do not
need to be handled in a symmetrical way. Because you set up a
supervision tree at boot time does not mean you need to get rid of it
at shutdown time.

  When you boot, you're starting from *nothing*, and you need to build
up to a state where the machine is functional and able to run services,
that's why it's incremental and deliberate and elaborate. It is
literally a bootstrap process that needs precise ordering.

  When you shutdown, you're starting from a state where everything is
already working, so you can rely on many more features, and you're not
trying to build anything, you're just trying to ensure consistency of
permanent state (aka disk) before you pull the plug. It's the only
thing that matters. If you were only ever operating in RAM, starting
from ROM / ro disk, and never had any mutable permanent state, your
shutdown procedure could just be an immediate reboot(), or pressing
the power off button; it is the case for some embedded devices. But
with mutable permanent state, we need to be more careful, that's why
we shut down services in an ordered way, and then make sure we can
unmount the filesystems before powering off. It's the *only* reason;
apart from that, you can do whatever you want. Who cares? the system is
going to be down anyway. When the Armageddon comes, you want to make
sure the time capsules are well sealed and buried for the next
civilization to find, but you don't have to clean your room.

  So the goals of boot and shutdown are very different.

  I specifically designed s6-linux-init so it would not store any vital
information in permanent mutable state, and would not hold any writing
fd on a filesystem. That means s6 will not prevent you from unmounting
your filesystems, parking your disks, whatever - and that the
supervision tree can keep running until you pull the plug. It is
designed to help you while the machine is running, and *especially*
in delicate situations where you're killing things and want to be sure
you can recover if something goes wrong rather than brick the system.
That's why I'm saying it's less effort to keep it in place and work
with disabling supervision when it needs to be disabled, than to
dismantle the supervision tree and have to reimplement recovery logic.


>And accidentally killing PID 1 would break the supervision tree based approach too, right?

  My bad, "killing pid 1" is the wrong wording, because it's actually
impossible. I meant: pid 1 exiting. *That* is something you need to
worry about when scripting, and that won't happen if you just keep
the supervision tree.


>But getting a more advanced recovery method up,
>e.g. an ssh server when you only have remote access
>or an agetty instead of PID 1 sh on a desktop machine,
>will be more reliable under a supervision tree, I can see that.

  Exactly.


>| if -n { mount -o remount,ro / }
>| foreground { s6-svc -U /run/service/recovery }

  You just tore down the supervision tree, and you want to start a
recovery... service? :D


>Anyhow, you have convinced me that keeping the supervision tree is the better way.

  \o/ This might be the first time someone listens to me \o/


--
  Laurent

Re: Scripting Stage 3 and 4

[Guillermo]

Hello,

El jue, 9 ene 2025 a las 17:25, Paul Sopka escribió:
>
> I figured that other people might also have some interest in this,
> so I am posting the question here.

May I ask if there is a purpose to this other than doing it as a
learning experience (if so, then by all means carry on)? Do you see a
problem with just using current s6-linux-init?

> Now I was wondering why you, Laurent recommend against using the finish
> script of s6-svscan
> for the shutdown procedure (here:
> https://skarnet.org/software/s6/s6-svscan-1.html#stage3).
>
> Using that would make good portion of my implementation more elegant and
> simple, [...]

Perhaps you already know, but if you are interested in comparing, one
can still download s6-linux-init-0.4.0.1 from the skarnet.org website
for studying, which is the last version that used an execline stage1
init, performed the shutdown procedure in .s6-svscan/finish and
shipped small C programs that probably did the same thing that
linux-powertool does here, before the change in package design —which
also surprised me a the time—. Although s6 dropped in version 2.10.0.0
the functionality that allowed those to work, and you'd need
skalibs-2.8.0.1 to be able to actually build and run that version of
s6-linux-init-maker and s6-{halt,poweroff,reboot}.

G.

Re: Scripting Stage 3 and 4

[Paul Sopka]

[-- Attachment #1.1.1.1: Type: text/plain, Size: 4200 bytes --]

>  With no tty? and not even a redirection from/to /dev/console? That's
> going to be a tough recovery 😉
I took redirection from/to /dev/console for granted xD, after all what I 
wrote as only part of a larger script.

>  And sure, you could wrap all your commands in a big if block. But my
> point is that these are *things you need to think about*. I'd rather
> not have to think about new things at shutdown time when I don't have
> to. Why change something and duplicate logic when you could just do
> nothing and keep what's already working and has worked for the whole
> lifetime of the machine? 
So you mean rather than (to quote myself):
> | if -n { mount -o remount,ro / }
> | foreground { s6-svc -U /run/service/recovery } 
You would just not tear down one tty (e.g. 12) or a "minimal recovery 
ssh server listening on an unusual port"?

>  The key to understanding how to manage a shutdown, I think, is to
> realize that boot and shutdown *are not symmetrical*, and so, do not
> need to be handled in a symmetrical way. Because you set up a
> supervision tree at boot time does not mean you need to get rid of it
> at shutdown time.
>
>  When you boot, you're starting from *nothing*, and you need to build
> up to a state where the machine is functional and able to run services,
> that's why it's incremental and deliberate and elaborate. It is
> literally a bootstrap process that needs precise ordering.
>
>  When you shutdown, you're starting from a state where everything is
> already working, so you can rely on many more features, and you're not
> trying to build anything, you're just trying to ensure consistency of
> permanent state (aka disk) before you pull the plug. It's the only
> thing that matters. If you were only ever operating in RAM, starting
> from ROM / ro disk, and never had any mutable permanent state, your
> shutdown procedure could just be an immediate reboot(), or pressing
> the power off button; it is the case for some embedded devices. But
> with mutable permanent state, we need to be more careful, that's why
> we shut down services in an ordered way, and then make sure we can
> unmount the filesystems before powering off. It's the *only* reason;
> apart from that, you can do whatever you want. Who cares? the system is
> going to be down anyway. When the Armageddon comes, you want to make
> sure the time capsules are well sealed and buried for the next
> civilization to find, but you don't have to clean your room.
>
>  So the goals of boot and shutdown are very different.
>
>  I specifically designed s6-linux-init so it would not store any vital
> information in permanent mutable state, and would not hold any writing
> fd on a filesystem. That means s6 will not prevent you from unmounting
> your filesystems, parking your disks, whatever - and that the
> supervision tree can keep running until you pull the plug. It is
> designed to help you while the machine is running, and *especially*
> in delicate situations where you're killing things and want to be sure
> you can recover if something goes wrong rather than brick the system.
> That's why I'm saying it's less effort to keep it in place and work
> with disabling supervision when it needs to be disabled, than to
> dismantle the supervision tree and have to reimplement recovery logic. 
Very interesting, thank you very much for this detailed response, I 
fully agree!
I think it might be worth putting this on the page I cited in the first 
mail.

>> | if -n { mount -o remount,ro / }
>> | foreground { s6-svc -U /run/service/recovery }
>
>  You just tore down the supervision tree, and you want to start a
> recovery... service? :D 

This was meant for the approach of keeping the supervision tree,
to be ran by the shutdown service.
But now it seems to me that it would be another case of

> *things you need to think about*
So, I guess it would be better to have a recovery service right away and 
not stop it,
like I suggested at the start of this mail?

Finally, wouldn't at least one "if" be appropriate,
to be sure whether everything is unmounted properly?


Regards,

Paul


[-- Attachment #1.1.1.2: Type: text/html, Size: 6875 bytes --]

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3195 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]

Re: Scripting Stage 3 and 4

[Paul Sopka]

[-- Attachment #1.1.1: Type: text/plain, Size: 1738 bytes --]

> May I ask if there is a purpose to this other than doing it as a
> learning experience (if so, then by all means carry on)? Do you see a
> problem with just using current s6-linux-init?
Of course you may ask!
Initially I wanted to learn and understand a proper (init/)shutdown 
procedure of UNIX like OS's.
But I figured, that if the scripted procedures turn out elegant and 
robust enough,
which - in my opinion - seems to be the case (again everybody please 
point out any flaws you find),
I would like to use them over s6-linux-init because

a) The scripts are very easy to port to other UNIX like OS's.
b) I am a big fan of 1 tool 1 job combined in a script.
c) The scripts are easier to understand (and modify) for sysadmins with 
little programming background.

I see no problem at all in s6-linux-init, it has worked 100% reliable on 
my machine for half a year now.

> Perhaps you already know, but if you are interested in comparing, one
> can still download s6-linux-init-0.4.0.1 from the skarnet.org website
> for studying, which is the last version that used an execline stage1
> init, performed the shutdown procedure in .s6-svscan/finish and
> shipped small C programs that probably did the same thing that
> linux-powertool does here, before the change in package design —which
> also surprised me a the time—. Although s6 dropped in version 2.10.0.0
> the functionality that allowed those to work, and you'd need
> skalibs-2.8.0.1 to be able to actually build and run that version of
> s6-linux-init-maker and s6-{halt,poweroff,reboot}.
Thanks for the heads up!
I did not know about that, that was before I discovered skarnet.
I will definitely look into that.

Regards,

Paul


[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3195 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 840 bytes --]