Re: s6-rc live state and database format upgrades

["Laurent Bercot" <ska-supervision@skarnet.org>]

>* s6-rc-format-upgrade was called with a compiled database that is not
>the exact 0.4.0.x equivalent of the one currently associated with the
>live state directory (i.e. other that a database created with
>s6-rc-compile from version 0.4.0.x and the exact same service
>definitions).

  It would probably mess up your live state and your dependencies, badly.
You would need to manually reboot to fix it.


>* s6-rc-update from version 0.4.0.x was called with a live state
>directory currently associated with a <=0.3.0.0 database.

  The update would not work: 0.4.0.x s6-rc, invoked from s6-rc-update,
would choke on your old database and exit with an error message.


>Is any of these things capable of trashing s6-rc's live state? I think
>that the documentation is clear about the upgrade procedure, but that
>these could be likely ways of accidentally screwing it up. Something
>like the latter was actually the database upgrade procedure for
>previous backwards-incompatible s6-rc upgrades, right? Although I
>don't know if any of them involved a database format change.
>
>Also, I'm lost about the role of version 0.3.0.1:

  What happened, in summary:

  * 0.4.0.x is a database format change over 0.3.0.x. The 0.3 format
implements pipelines, but the 0.4 format implements funnels instead.

  * The simplest way of upgrading the format is just... to put the
new database in place of the old one, without any conversion whatsoever.
That is what the 0.4.0.x s6-rc-format-upgrade does. It blindly replaces
the $livedir/compiled symlink with a symlink to the new database. It
does zero magic: the magic happens earlier. (But you should not rely on
this, because s6-rc-format-upgrade MAY do magic in the future.)

  * Simple replacement can only work iff the new db and the old db
represent the same set of services AND those services have the same
numbers. IOW: the old db contents and the new db contents are exactly
the same, only the format changes.

  * Up to 0.3.0.0, s6-rc-compile does not guarantee the numbers for the
services - it assigns the numbers as it finds the source definition
directories with readdir(). So it's impossible to make sure that a
0.4.0.x db can be a drop-in replacement.

  * That is why I released 0.3.0.1, where s6-rc-compile actually uses
deterministic numbering of its services: the same source db will always
produce the same compiled db. (It's just a question of adding a
qsort() when you have all the service names in a source directory. :))
This is not official or part of the public API: users aren't supposed to
know or use service numbers, or rely on deterministic numbering.

  * 0.4.0.x uses the same deterministic numbering. So, the same source
db compiled with 0.3.0.1 and 0.4.0.x will produce the same compiled
databases with only the format changing, and those dbs are suitable
to be exchanged with s6-rc-format-upgrade.

  * The 0.3.0.1 stepping stone was only necessary because prior to it,
service numbering was not deterministic. Now that it is deterministic,
the intermediary step will not be necessary for future format upgrades.
IOW: you will be able to switch directly from 0.4.0.x to 0.5.0.0. Or
from 0.3.0.1 to 0.5.0.0.


>Development of 0.3.0.1 was done in the skarnet.org Git repository in a
>'real_0.3.0.1' branch. Development of 0.4.0.0 was done in the master
>branch

  Yes, because I first worked on 0.4.0.0, in the master branch, and
then realized that a format change would break people's live states,
and a clean upgrade path was needed. So I wrote s6-rc-format-upgrade
and soon realized that dealing with arbitrary changing service numbers
was a nightmare, and I could make my life a lot easier if the service
numbers were deterministic. But to ensure a proper upgrade, they also
needed to be deterministic *before* the upgrade - so a 0.3.0.1 release
was necessary, but the master branch was 0.4.0.0, so I created a
branch to work on 0.3.0.1.


>  and the current upgrade notes don't even mention 0.3.0.1:

  That is my bad: I didn't backport the 0.3.0.1 lines to the master 
branch.
Thanks for the report, will fix.


>So, in the end, does upgrading from 0.3.0.0 to 0.4.0.x require an
>intermediate 0.3.0.1 s6-rc-compile + s6-rc-update step, and then
>0.4.0.x s6-rc-compile + s6-rc-format-upgrade, or not?

  Yes, it does! You can only upgrade to 0.4.0.x from 0.3.0.1. Sorry
for not making it clearer, and sorry for the inconvenience; later
format upgrades will be simpler.


>(I didn't have running services that I had to preserve, so personally
>I didn't need any special care and just upgraded from whatever I had
>at the time to s6-rc-0.4.0.0, but s6-rc-format-upgrade is now there
>and one might need to explain or remind correct usage :) )

  If you don't mind breaking things and manually rebooting, you can
of course just yolo the db change without bothering with intermediary
steps. :)

--
  Laurent