A PkgSrc VM for Helios / Omicron1

A PkgSrc VM for Helios / Omicron1

[d]

Is it possible, and how hard would it be to make a fork of the omicron1 
branded sparse VM for Helios that supports PkgSrc?


The background:

I'm planning a fork of Helios, Oxide Computer's fork of Illumos that 
makes it possible to run their cloud stack on normal servers. This has 
been made a little easier since they have an dev system based on OmniOS 
that they use for development.

Although Bhyve provides a world of choices for VM's, I think I'd rather 
run a sparse container that uses PkgSrc.

Is this a good idea?

Any advice or suggestions?

Daniel
SolOS-Cloud Discord: https://discord.gg/MGTXaGfCUf

Re: [discuss] A PkgSrc VM for Helios / Omicron1

[Joshua M. Clulow]

On Sun, 4 Aug 2024 at 08:07, d <omnios@puptv.com> wrote:
> Is it possible, and how hard would it be to make a fork of the omicron1
> branded sparse VM for Helios that supports PkgSrc?

The source for the omicron1 brand is here:

    https://github.com/oxidecomputer/helios-omicron-brand

It's not completely accurate to say that it's sparse, for what it's
worth.  Sparse zones currently mount some trees, like /usr, read-only
from the host system.  You then can't modify that tree at all; what's
in the GZ version of /usr is in the zone, no more, no less.

I wanted to add the ability to modify the contents of /usr, to be able
to add or remove files as part of layering on additional packages -- a
bit like a docker image does.  We don't currently have a union file
system, and in fact union file systems are complex in (correct)
implementation, which is why I didn't just write one at the time.
Instead, I used some important facts about the intended structure of
Helios systems in the Oxide product (a ramdisk-based appliance) to
make a simpler approach.

At zone install time, we:

    - create a blank ZFS file system
    - unpack a baseline tar file to get a blank /etc and /var
    - make a recursive _copy_ of whatever happens to be in /usr and
      /lib in the GZ into the zone file system
    - unpack more layer tar files on top of the result of the
      previous steps, if provided

Both the base ramdisk image for the host, and the baseline tar file,
are assembled together from IPS packages at image build time.  This
means they're never out of sync with one another.  Also critically,
libc in the zone image is never out of date with the host kernel, even
though we made a copy, because it's all ramdisk ephemera anyway; when
we reboot and want to start more zones, we install them from scratch,
potentially mounting in delegated datasets that have the specific
persistent data we needed (like database files, etc).

This presents a challenge, though, for workstations and engineering
systems in the lab, where we use a classic install-to-disk UNIX system
approach that allows one to add packages and mutate the system in the
ways that are convenient in that setting.  There, each time you "pkg
update" to get new OS software, the kernel _would_ be out of sync with
the libraries zone root.  The baseline archive would also be out of
sync, and no package update actions would have occurred.  We solve the
baseline issue by regenerating it in-situ using pkg(1) in a boot-time
SMF service after each boot.

As described in the omicron1(7) manual page, and the README, for
development systems it's your responsibility to hold it correctly.
This means any time you do a "pkg update" and reboot, you must also
uninstall and reinstall any zones after you reboot.  This will use the
updated baseline, and will also re-copy all the files from /usr that
must be in sync.  Ultimately, you must simulate what we do on the
ramdisk system for it all to hold together in the expected way.

I don't have a succinct answer to the question, but there are some
things that might help you answer it on your own:

    - the pkgsrc bootstrap is just files to lay down in the file
      system, so really you could probably just create a regular
      omicron1 brand image layer tar file that you could pass
      to the regular (unmodified) brand to get a zone that has
      pkgsrc bits inside, just as we do with Omicron components

    - the omicron1 brand is really, for production use, intended
      to be used in a very particular way, in a very particular
      ramdisk environment (described in the documentation and
      above) -- if you aren't planning on building and booting
      ramdisk images I'm not sure that I would use it, given
      the caveats for its use on install-to-disk systems

    - we made a conscious choice to focus on hardware-level
      virtualisation in the Oxide product, in no small part due
      to the (relative!) ease with which user workloads can
      be punted between machines using live migration; this is
      not something that's likely to be easy or even possible
      to achieve with zone-based workloads -- if you're planning
      to use our control plane, but on i86pc systems, I suspect
      you're going to run into more and more friction as we
      lean more heavily into the live migration stuff for
      things like seamless software update and so on.


Cheers.

-- 
Joshua M. Clulow
http://blog.sysmgr.org