[TUHS] Unix install & "standalone" package

[TUHS] Unix install & "standalone" package

[Norman Wilson]

I don't remember any special many-programs-in-one binary
like busybox in any Unix from the days when Unix was simple
enough for me to understand.  That covers the entire lifetime
of the Research systems, but also System V and the BSDs and
their sundry offspring up into at least the 1990s.

I'm pretty sure OpenBSD at least still has nothing like
busybox.  The nearest thing was to make sure certain programs
were linked statically (or existed in alternate statically-
linked versions) so they would work before shared libraries
were available.  (It seems to be common wisdom that `sbin'
means `system bin' these days, but I remember once, long ago,
being told it stood for `static bin'.)

Perhaps the question to ask is why such a magic program is
needed at all.  Is it just because programs like the shell
have become so large and unwieldy that they won't fit in
a small environment suitable for loading into an initramfs?

Older UNIXes, even on the VAX, didn't use an initramfs.
the boot code had just enough understanding of devices and
file systems to load the kernel and to point it at the
real root file system.  The VAX hardware designers had
a clever scheme on many (but, strangely, not all) VAX
variants by which the hardware had several little boot
ROMs, each containing a bare-bones read-only device
driver for a particular device along with a few
instructions to read the first sector into memory
and start it, with pointers to the boot-rom driver
and device ID in specified registers.  That was enough
to support a device-independent Unix boot block that
could read unix (or another file name typed on the
console) from the root of the file system at the start
of the disk.  I know it was because I wrote such a
boot block, though I don't know whether anyone else
did.  (Other systems, I think, just used it to load
/boot, a larger and more-capable program.)

Maybe it was just that the boot environment was simpler
in older systems, without the need to load kernel modules
or support multiple locations and means of access for
the root?

Norman Wilson
Toronto ON

[TUHS] Re: Unix install & "standalone" package

[Vincenzo Nicosia]

rather, back then you didn't need the same kernel to run on a wide
variety of hardware, with all the possible different combinations of
peripherals, requiring all sorts of different drivers. 

I think that's the only real reason why initramfs came to existence:
allowing a selection of kernel modules to be loaded at init time,
depending on the hardware at disposal on that machine. Then things went
south, and more recent initramfs have everything and the kitchen sink.
But that's another story.

HND

Enzo Nicosia

-- 

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Mon, Sep 4, 2023 at 8:56 AM Vincenzo Nicosia <katolaz@freaknet.org>
wrote:

> rather, back then you didn't need the same kernel to run on a wide
> variety of hardware, with all the possible different combinations of
> peripherals, requiring all sorts of different drivers.
>
> I think that's the only real reason why initramfs came to existence:
> allowing a selection of kernel modules to be loaded at init time,
> depending on the hardware at disposal on that machine. Then things went
> south, and more recent initramfs have everything and the kitchen sink.
> But that's another story.


Yea, it was an effort to move mounting of root out of the kernel. The
earliest
scripts just mounted the right disk and moved on, and didn't load any new
drivers: they just had the logic to pick the desired root. But at the same
time,
there were a lot of people that were running on 4MB and 8MB  systems that
noticed they could put all the router software in the initramfs and never
pivot
to something else and they could have quite the product with that. And those
were the first few bricks that paved the road to hell :)

Warner

[-- Attachment #2: Type: text/html, Size: 1584 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Clem Cole]

[-- Attachment #1.1: Type: text/plain, Size: 4871 bytes --]

I can not add too much to what happened later, but I think modern users
don't get it why the things like the standalone system were needed back in
the day.  There were a few simple differences thanks to Moore's law, modern
folks really may not grok:

   1. The number of peripherals that needed to be supported was small [each
   firm made only a few for each target system]
   2. The capacity/capabilities of those peripherals are minimal compared
   to today's world
   3. Cheap ROMs, as we know it, were coming on the scene [the EPROM is
   only invented in 1971] One-time programmable and Mask ROMs were around but
   they were expensive and small.


I hope Warren's emailer lets this pass through.

Here is a picture of the original root 'ROM' for the PDP-11 M792-xx board,
which gives you 32 words (64 bytes) of memory.  It used individual diodes
in or out arranged an array to represent individual bits with a diode
present being a '1' and '0' if missing.  As the Gunkies web page suggests,
the board came in two versions - customer programmable which was fully
populated with diodes, and the user removed unneeded diodes to program it.
The other variants were programmed with the designed boot functions -
designated M792-Yx (where 'x' is a capital letter, starting with 'A')

[image: Read_Only_Memory_DEC_M792_Diode_Matrix_ROM_Aceware_Attribution.jpg]


Also,  remember that a $3-5k/drive RK05 is only 2.5Mbytes [4872 512 byte
'sectors' *a.k.a*. disk blocks -- 2 hd/disk * 15 sec/cyl * 203 cyl/disk],
which is why we had separated /bin and /usr/bin and /lib and /usr/lib.
Plus, as Norman points our dynamic linking only comes later, so all of
these are static bound programs (*i.e.,* Sun create /bin and /sbin to
separate those programs).  The whole idea was what was in root was enough
to get the system running in /bin, /lib and /etc and everything else was
one the mounted file system, which often was another RK05.   Many of our V6
and V7 systems ran with 3 RK05 /, /usr /home, and we might have an extra
RK05 for /mnt.  Tape was the standard thing.   DEC tape was cheaper than
9-track, although in my history, we all went for 9-track because it was
more portable to other systems, but often used after-market 9-track
transports that emulated the DEC ones.

Even with the VAX, the front-end runs on an LSI-11 with floppies, so things
like microcode were loaded into the VAX and stored in ROM.   By then, the
RP and RM series had become more normal, and eventually, the RK07 and RL02
replaced the RK05.   But we are not talking about substantial capacity in
the storage systems.   Only ten years later, in the early 1980s, a
Fujisu 'Eagle' is just 470Mbytes and costs 12-15K per drive, plus another
$6-8K for the controller for your Vax [also often after-market from SI or
Emulex].

Sun and Apollo build 'diskless' systems because a 100MB ST-506 style disk
was considered too expensive, and was worried about the entry-level price
of their systems (I famously made a typo describing how those systems
performed BTW with a dyslexic change of the s to c on a whole company email
at Masscomp in the mid-1980s - to Sun's credit, it was the best marketing
ploy ever - people bought diskless and discovered they sucked and then all
added a disk - which cost more $s than the original Masscomp system which
was had one built-in).  But I digress...

Those new workstations did have real ROM chips, unlike the original
PDP-11s, but they were small, maybe 8Kbytes of NS2764, so the ROMs were
tight and only had enough in them to recognize a couple of peripherals.

So to answer more of your question, you build the tools you need that make
sense at the time.   /stand was a simple solution and was small.   For V7
and the original BSD 3 and 4 released, the user needed to read a
distribution media, traditionally a 9-track tape, and set up a local disk.
The number of peripherals was small.

One other thought ... in the case of the PDP-11 and Vaxen, the disks were
often dismountable, like the different RK, RL, RP, and RM series drives.
 We often had more than one system.  So once the system was up and running,
it was not usual to use a different system with better tools to help repair
things if bad stuff occurred.  That is also why we often used 9-track tape,
just because that was even easier to move back and forth.  With the
Wokstrations, the disks were generally sealed.

Clem

A  small PS on another note -- I have memories of taking an ST-506 disk
with me to France to work with some engineers in Grenoble and explaining to
the French authorities that they could not open it up to look inside.
 BTW: this pre-internet - imagine what we had to do if we flew to Canada,
which was taxing SW.   Even bringing tapes in and out of Toronto airport
could be treacherous.

ᐧ

[-- Attachment #1.2: Type: text/html, Size: 5769 bytes --]

[-- Attachment #2: Read_Only_Memory_DEC_M792_Diode_Matrix_ROM_Aceware_Attribution.jpg --]
[-- Type: image/jpeg, Size: 154411 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Paul Winalski]

On 9/4/23, Clem Cole <clemc@ccc.com> wrote:
>
> Even with the VAX, the front-end runs on an LSI-11 with floppies, so things
> like microcode were loaded into the VAX and stored in ROM.

I think you meant either "stored in RAM" or "not stored in ROM".  The
LSI-11 front end for the VAX-11/780 (and other early VAXen?) was
connected to the SBI (Synchronous Backplane Interconnect--the system
bus) and performed three main functions:  microcode load and system
bootstrap initiation, interface for the system console (originally a
LA-36 dot matrix terminal), and some system diagnostics.  CPU
microcode was stored on a 5" floppy disk connected to the LSI-11 and
was loaded into RAM at system power-up.

-Paul W.

[TUHS] Re: Unix install & "standalone" package

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Tue, Sep 5, 2023 at 1:03 PM Paul Winalski <paul.winalski@gmail.com>
wrote:

> On 9/4/23, Clem Cole <clemc@ccc.com> wrote:
> >
> > Even with the VAX, the front-end runs on an LSI-11 with floppies, so
> things
> > like microcode were loaded into the VAX and stored in ROM.
>
> I think you meant either "stored in RAM" or "not stored in ROM".

Yep -- thanks - dyslexia-r-me.


ᐧ

[-- Attachment #2: Type: text/html, Size: 1330 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Theodore Ts'o]

On Mon, Sep 04, 2023 at 11:20:31AM -0600, Warner Losh wrote:
> 
> Yea, it was an effort to move mounting of root out of the
> kernel. The earliest scripts just mounted the right disk and moved
> on, and didn't load any new drivers: they just had the logic to pick
> the desired root. But at the same time, there were a lot of people
> that were running on 4MB and 8MB systems that noticed they could put
> all the router software in the initramfs and never pivot to
> something else and they could have quite the product with that. And
> those were the first few bricks that paved the road to hell :)

The other reason why you might need something like an initial ramdisk
is if you don't yet *have* a root file system on the hardware, and you
are trying to install a system using TFTP boot.  At that point, you
have three choices:

(a) Use a NFS root

(b) Use a read-only network block device (for example, MIT Project
Athena had a read-only Remote Virtual Disk which was a network block
device that was used to provide a read-only system image used for
installation and update, as well as a read-only /usr image).

(c) Use a built-in ramdisk.  (And here's not just Linus which has
adopted this strategy; OpenBSD's installer does this as well.)


The other dynamic at play was that ramdisks were needed when you were
installing on a 386 PC with, say, 16 megs of memory, a 320 megabyte
hard disk, and a *single* 1.44MB floppy --- and no ethernet, because
this was for the home PC user, so the best that you had might be a
38kbps dialup link with PPP --- if you were lucky.

So putting the kernel on the first floppy disk, and putting a ramdisk
image on the second floppy disk, so you could then eject the floppy
disk and insert subsequent floppy disks so you could install rest of
the installation binaries (including the shell utilities, the C
compiler, X windows systems, etc.) and then reboot onto the HDD with a
fully set up system, was a pretty natural evolution.

And since we had the ramdisk infrastructure, and most users didn't
want to configure their own kernel as part of the installation
process, that begat kernel modules and the initial ramdisk being used
to store the kernel modules.  In the very early days, this made Linux
*far* more user friendly for non-system programmers to install,
compared to say, FreeBSD during that era, which was still stuck in the
BSD 4.x days of a generic kernel, followed by a kernel configuration
step, etc.

These days, given that some enterprise setups what their servers to be
installed from a Fibre Channel Storage Area Network, or iSCSI, perhaps
using Kerberos to authenticate to the iSCSI target, it makes
absolutely a huge amount of sense to have an initial ramdisk as an
option.  That being said, for a long time, the initial ramdisk was an
*option*.  You didn't have to use it, if you were willing to have a
custom kernel and you are using a device which has a fixed and stable
boot-time enumeration.

This tended to only be used by people who knew what they were doing,
but for example, my file system test appliance VM system doesn't use
an initial ramdisk at all.  It uses qemu, and has a built-in kernel
configuration and build system, and this works perfectly fine booting
into the latest Debian stable distribution, with no initial ramdisk
used at all:

https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md

						- Ted





the root filesystem on the second floppy disk (the first
floppy disk contained the kernel), which was then copied into a
ramdisk, so you could 


Trying to fit the kernel and the root file system on a single 1.44MB
floppy, was (barely) doable, but you certainly wouldn't have space for
a C compiler so you build or link your own custom kernel.  So one of
the first things that we did was to put the kernel on a single floppy
disk, and then but the root on a single floppy disk, and then copy the
root to the HDD, and then reboot onto the HDD, and then take the
installation procedure from there.  But that was awkward and 

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 5093 bytes --]

On Mon, Sep 4, 2023 at 1:59 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Mon, Sep 04, 2023 at 11:20:31AM -0600, Warner Losh wrote:
> >
> > Yea, it was an effort to move mounting of root out of the
> > kernel. The earliest scripts just mounted the right disk and moved
> > on, and didn't load any new drivers: they just had the logic to pick
> > the desired root. But at the same time, there were a lot of people
> > that were running on 4MB and 8MB systems that noticed they could put
> > all the router software in the initramfs and never pivot to
> > something else and they could have quite the product with that. And
> > those were the first few bricks that paved the road to hell :)
>
> The other reason why you might need something like an initial ramdisk
> is if you don't yet *have* a root file system on the hardware, and you
> are trying to install a system using TFTP boot.  At that point, you
> have three choices:
>
> (a) Use a NFS root
>
> (b) Use a read-only network block device (for example, MIT Project
> Athena had a read-only Remote Virtual Disk which was a network block
> device that was used to provide a read-only system image used for
> installation and update, as well as a read-only /usr image).
>
> (c) Use a built-in ramdisk.  (And here's not just Linus which has
> adopted this strategy; OpenBSD's installer does this as well.)
>

Yes. FreeBSD did this originally, and some installers still do that.


> The other dynamic at play was that ramdisks were needed when you were
> installing on a 386 PC with, say, 16 megs of memory, a 320 megabyte
> hard disk, and a *single* 1.44MB floppy --- and no ethernet, because
> this was for the home PC user, so the best that you had might be a
> 38kbps dialup link with PPP --- if you were lucky.
>
> So putting the kernel on the first floppy disk, and putting a ramdisk
> image on the second floppy disk, so you could then eject the floppy
> disk and insert subsequent floppy disks so you could install rest of
> the installation binaries (including the shell utilities, the C
> compiler, X windows systems, etc.) and then reboot onto the HDD with a
> fully set up system, was a pretty natural evolution.
>

Yes. FreeBSD 2.0 definitely did that, and I think FreeBSD 1.0 as well.


> And since we had the ramdisk infrastructure, and most users didn't
> want to configure their own kernel as part of the installation
> process, that begat kernel modules and the initial ramdisk being used
> to store the kernel modules.  In the very early days, this made Linux
> *far* more user friendly for non-system programmers to install,
> compared to say, FreeBSD during that era, which was still stuck in the
> BSD 4.x days of a generic kernel, followed by a kernel configuration
> step, etc.
>

GENERIC worked fine in the early days: it was small enough that the savings
from removing all the extra devices was small... but that's because the
number
of devices supported was also relatively small. By the time it became an
issue
FreeBSD did have dynamic loading of drivers, but retained the GENERIC
kernel for install (it should have gone to a minimal kernel, but hasn't yet
completed the migration: the savings wasn't big enough due to Moore's
Law growing machine memories faster than the kernel grew: FreeBSD's kernel
only doubled every 2 years...).

But yea, it was a bit of a kludge. And FreeBSD grew the ability to configure
the morass of ISA devices in the boot loader early on so you could at least
boot the generic kernel, and have the loader remember your choices.


> These days, given that some enterprise setups what their servers to be
> installed from a Fibre Channel Storage Area Network, or iSCSI, perhaps
> using Kerberos to authenticate to the iSCSI target, it makes
> absolutely a huge amount of sense to have an initial ramdisk as an
> option.  That being said, for a long time, the initial ramdisk was an
> *option*.  You didn't have to use it, if you were willing to have a
> custom kernel and you are using a device which has a fixed and stable
> boot-time enumeration.
>

Yea. FreeBSD can use it to this day (and many interesting use cases
use it), but generally the loader tells the kernel what root filesystem to
use. These smarts in the boot loader have also kept the pressure of the
vast majority of cases where initramfs makes sense in Linux land.


> This tended to only be used by people who knew what they were doing,
> but for example, my file system test appliance VM system doesn't use
> an initial ramdisk at all.  It uses qemu, and has a built-in kernel
> configuration and build system, and this works perfectly fine booting
> into the latest Debian stable distribution, with no initial ramdisk
> used at all:
>
>
> https://github.com/tytso/xfstests-bld/blob/master/Documentation/kvm-quickstart.md
>

Ah, cool. Every single distro I've run in the past has had an initramfs. I
thought
it had become no longer an option...

I do similar things for my VM setup with FreeBSD :)

Warner

[-- Attachment #2: Type: text/html, Size: 6738 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 2061 bytes --]

On Mon, Sep 4, 2023 at 8:44 AM Norman Wilson <norman@oclsc.org> wrote:

> Maybe it was just that the boot environment was simpler
> in older systems, without the need to load kernel modules
> or support multiple locations and means of access for
> the root?
>

Older systems had fewer choices. On the PDP-11 you had Q-Bus or Unibus.
You booted off a small selection of disks that converged to MSCP(?), so
there was THE boot environment and THE driver and THE filesystem. And
the kernels were often tuned to be exactly what the machine needed.

The VAX, Sun, hp, ibm, etc continued this early trend. Though  the boot
process
was still fairly narrow, with a limited list of supported boot devices.

In large part this was because the machine was exactly the same every
time you booted.

But with USB, PC Card, CardBus, ExpressCard, SCSI, Thunderbolt,
and a host of other removable technologies with a dizzying array of
cards, like PCI, ISA, etc, the game changed.

Also, the number of environments that a FreeBSD GENERIC[*] kernel
can boot in is huge due to combinatoric explosion due to three or
four boot environments, vm alternate startup paths, several supported
root file system, BIOS, UEFI, OpenFirmware, u-boot, coreboot, etc
as well as several thousand supported boot controller devices, things
got complex, despite there being several common interfaces that made
things simpler. I'm running a summer of code project this year to help
tame the combinatoric explosion to provide better test coverage because
though attempts were made to make things the same, variations exist that
can cause unexpected breakage in different environments.

Warner

[*] I use GENERIC here as a catch all, including the more recent MINIMAL
kernels that try to include just the core functionality, and omit the vast
majority of drivers. Linux has similar issues, but more, and also solves
them in many interesting ways... But I'm the FreeBSD boot loader guy
in large part (though there's others that work on it), not a Linux person.

[-- Attachment #2: Type: text/html, Size: 2719 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Steffen Nurpmeso]

Norman Wilson wrote in
 <9A989054DE79CE5059CBA74797391E39.for-standards-violators@oclsc.org>:
 |I don't remember any special many-programs-in-one binary
 |like busybox in any Unix from the days when Unix was simple
 |enough for me to understand.  That covers the entire lifetime
 |of the Research systems, but also System V and the BSDs and
 |their sundry offspring up into at least the 1990s.
 ...
 |Perhaps the question to ask is why such a magic program is
 |needed at all.  Is it just because programs like the shell
 |have become so large and unwieldy that they won't fit in
 |a small environment suitable for loading into an initramfs?

AlpineLinux as used on my vserver has busybox by default and can
cover most utitilities like that.  The lead developer Copa once
said something like "The idea is you install explicitly [if you
want something better]".  (It is a symlink farm that is
selectively replaced by installing "real" packages iirc.)

For my laptop it allows me easy boot management.
To save you the chatter ("Chatten" is the name of my tribe .. most
likely; could be Franken, Sueben .. and you know how it is): this
approach is much easier and smaller than having lots of static
binaries to copy around etc.

I do not use secure boot, i have on EFI only a kernel, busybox and
cryptsetup, and scripts (the laptop is named "kent")::

  ...
  drwxr-xr-x  4 root root     4096 Jul 15  2021  EFI/
  ...
  -rwxr-xr-x  1 root root      272 Feb  1  2022  kent.sh*
  -rwxr-xr-x  1 root root      313 Feb  1  2022  kent-direct.sh*
  drwxr-xr-x  1 root root      252 Oct  9  2022  ../
  -rwxr-xr-x  1 root root     4596 Feb  4  2023  linux-init-s1.sh*
  -rwxr-xr-x  1 root root     3646 Feb  4  2023  linux-init-lib.sh*
  -rwxr-xr-x  1 root root  5480120 Feb 11  2023  cryptsetup.static*
  -rwxr-xr-x  1 root root  1978368 Aug 15 18:51  busybox.static*
  -rwxr-xr-x  1 root root 10112672 Aug 26 18:44  ideapad-stage1.efi*

So kent.sh can be init(8) for the ideapad-stage1.efi Linux kernel
started via EFI as setup via efibootmgr(8) 

  Boot0001* kent HD(1,GPT,5d6d756b-5de2-4e5d-b043-8d4ae1bb6eb0,0x800,0x82000)/File(\ideapad-stage1.efi)root=/dev/nvme0n1p1 rootfstype=vfat init=/kent.sh

  #!/busybox.static sh
  #@ kent, step 1., via EFI.
  PART_ROOT=/dev/nvme0n1p8
  ROOT_DECRYPT='-t btrfs -o defaults,subvol=/crux/kent/root'
          PART_ROOT1=/dev/nvme0n1p8
          ROOT_DECRYPT1='-t btrfs -o defaults,subvol=/crux/kent/root.old'
  INIT_S2=/boot/kent-2.sh
  . /linux-init-s1.sh

and that allows me to unlock the harddisk.

We then boot via $INIT_S2 and kexec(8) a kernel from the encrypted
harddisk, so no code from EFI partition keeps on running.  (We
byte-compare the data from EFI with equal /boot/ files after
booting the real system.)

This allows nice and easy properties: only three files to track
(cryptsetup, busybox, kernel), almost same set of files in /boot/
and /media/efi aka EFI.  And ideapad-stage1.efi is the same kernel
that later runs, but later we have also additional dynamic modules
available.  Ie, every few weeks i copy /boot/ideapad-6_1.efi over
to be the new -stage1.efi.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: Unix install & "standalone" package

[Steffen Nurpmeso]

Steffen Nurpmeso wrote in
 <20230904221059.sF2G0%steffen@sdaoden.eu>:
 |Norman Wilson wrote in
 | <9A989054DE79CE5059CBA74797391E39.for-standards-violators@oclsc.org>:
 ...
 ||Perhaps the question to ask is why such a magic program is
 ||needed at all.  Is it just because programs like the shell
 ||have become so large and unwieldy that they won't fit in
 ||a small environment suitable for loading into an initramfs?
 ...
 |For my laptop it allows me easy boot management.
 ...
 |  -rwxr-xr-x  1 root root     4596 Feb  4  2023  linux-init-s1.sh*
 |  -rwxr-xr-x  1 root root     3646 Feb  4  2023  linux-init-lib.sh*
 |  -rwxr-xr-x  1 root root  5480120 Feb 11  2023  cryptsetup.static*
 |  -rwxr-xr-x  1 root root  1978368 Aug 15 18:51  busybox.static*
 |  -rwxr-xr-x  1 root root 10112672 Aug 26 18:44  ideapad-stage1.efi*

Only to add that this is because of Linux and the way it is doing
things.  If i would use FreeBSD on bare metal, then i would have
an EFI boot loader on EFI that knows (only) enough to ask for
passphrase (correct me if i am wrong), and can then boot the
kernel from FFS or ZFS.  (You have to choose dedicated ZFS boot
loader iirc, but despite that...)

I know GRUB (and maybe other) Linux bootloaders can do all that,
but they are huge, are badly maintained, or under-documented, let
alone with local manuals, and i am too stupid to configure them
(due to all that).  refind is ok, however.  But.. be aware of
typos in the configuration..

But anyhow.  With an EFI_STUB Linux kernel i can save me all that,
with busybox i get a complete environment (i then even create an
initrd in /boot/ on the fly so i do not have to type the password
a second time, that can (optionally) be cached, and is, actually

  -rw------- 1 root root  4495987 May 29 16:29 .kent.initrd.0

Unfortunately cryptsetup is needed even though, i think, the
kernel has anything needed; you just cannot access it.  cryptsetup
is only needed for "$cs open $PART_ROOT p_root --key-file -".
Of course i am no real Linux expert but only a do-it-yourself guy.
busybox allows me to manage this easily, to answer your question.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 3147 bytes --]

On Tue, Sep 5, 2023 at 9:53 AM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:

> Steffen Nurpmeso wrote in
>  <20230904221059.sF2G0%steffen@sdaoden.eu>:
>  |Norman Wilson wrote in
>  | <9A989054DE79CE5059CBA74797391E39.for-standards-violators@oclsc.org>:
>  ...
>  ||Perhaps the question to ask is why such a magic program is
>  ||needed at all.  Is it just because programs like the shell
>  ||have become so large and unwieldy that they won't fit in
>  ||a small environment suitable for loading into an initramfs?
>  ...
>  |For my laptop it allows me easy boot management.
>  ...
>  |  -rwxr-xr-x  1 root root     4596 Feb  4  2023  linux-init-s1.sh*
>  |  -rwxr-xr-x  1 root root     3646 Feb  4  2023  linux-init-lib.sh*
>  |  -rwxr-xr-x  1 root root  5480120 Feb 11  2023  cryptsetup.static*
>  |  -rwxr-xr-x  1 root root  1978368 Aug 15 18:51  busybox.static*
>  |  -rwxr-xr-x  1 root root 10112672 Aug 26 18:44  ideapad-stage1.efi*
>
> Only to add that this is because of Linux and the way it is doing
> things.  If i would use FreeBSD on bare metal, then i would have
> an EFI boot loader on EFI that knows (only) enough to ask for
> passphrase (correct me if i am wrong), and can then boot the
> kernel from FFS or ZFS.  (You have to choose dedicated ZFS boot
> loader iirc, but despite that...)
>

No, you don't have to choose the dedicated ZFS boot loader, at least not
anymore.

Also, you can use boot1.efi to load loader.efi from the root filesystem to
load the kernel, or you could use loader.efi directly on the ESP to load
the kernel. boot1 barely knows anything (and has only one choice of
what to boot). loader.efi is the full deal, and can do rather a lot of
sophisticated things.


> I know GRUB (and maybe other) Linux bootloaders can do all that,
> but they are huge, are badly maintained, or under-documented, let
> alone with local manuals, and i am too stupid to configure them
> (due to all that).  refind is ok, however.  But.. be aware of
> typos in the configuration..
>
> But anyhow.  With an EFI_STUB Linux kernel i can save me all that,
> with busybox i get a complete environment (i then even create an
> initrd in /boot/ on the fly so i do not have to type the password
> a second time, that can (optionally) be cached, and is, actually
>
>   -rw------- 1 root root  4495987 May 29 16:29 .kent.initrd.0
>
> Unfortunately cryptsetup is needed even though, i think, the
> kernel has anything needed; you just cannot access it.  cryptsetup
> is only needed for "$cs open $PART_ROOT p_root --key-file -".
> Of course i am no real Linux expert but only a do-it-yourself guy.
> busybox allows me to manage this easily, to answer your question.
>

You could do that on FreeBSD with a loader.efi that has a ram disk
built into it as well, including a 'beastie box' thing that's akin to
busybox.
It will boot in one step and no no further I/O to get a running system.
Others have used this for secure boot and to boot a small ram disk that's
later discarded as userland decides what root should be. But it's much less
automated than in Linux...

Warner

[-- Attachment #2: Type: text/html, Size: 4136 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Steffen Nurpmeso]

Hello!

Warner Losh wrote in
 <CANCZdfoy0UJyFdvJ_fO4FDks_dfEOhsJo+7p2RORdk6gQWf7qg@mail.gmail.com>:
 |On Tue, Sep 5, 2023 at 9:53 AM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
 |> Steffen Nurpmeso wrote in
 |>  <20230904221059.sF2G0%steffen@sdaoden.eu>:
 |>|Norman Wilson wrote in
 |>| <9A989054DE79CE5059CBA74797391E39.for-standards-violators@oclsc.org>:
 |>  ...
 |>||Perhaps the question to ask is why such a magic program is
 |>||needed at all.  Is it just because programs like the shell
 |>||have become so large and unwieldy that they won't fit in
 |>||a small environment suitable for loading into an initramfs?
 |>  ...
 |>|For my laptop it allows me easy boot management.
 ...
 |> Only to add that this is because of Linux and the way it is doing
 |> things.  If i would use FreeBSD on bare metal, then i would have
 |> an EFI boot loader on EFI that knows (only) enough to ask for
 |> passphrase (correct me if i am wrong), and can then boot the
 |> kernel from FFS or ZFS.  (You have to choose dedicated ZFS boot
 |> loader iirc, but despite that...)
 |
 |No, you don't have to choose the dedicated ZFS boot loader, at least not
 |anymore.

Ah!  It seems regarding FreeBSD i am stuck in BIOS world, then.
(zfsboot and gptzfsboot.)

 |Also, you can use boot1.efi to load loader.efi from the root filesystem to

The former deprecated i see.

 |load the kernel, or you could use loader.efi directly on the ESP to load
 |the kernel. boot1 barely knows anything (and has only one choice of
 |what to boot). loader.efi is the full deal, and can do rather a lot of
 |sophisticated things.

Great.

  ...
 |> But anyhow.  With an EFI_STUB Linux kernel i can save me all that,
 |> with busybox i get a complete environment (i then even create an
 |> initrd in /boot/ on the fly so i do not have to type the password
 |> a second time, that can (optionally) be cached, and is, actually
 ...
 |> Unfortunately cryptsetup is needed even though, i think, the
 |> kernel has anything needed; you just cannot access it.  cryptsetup
 |> is only needed for "$cs open $PART_ROOT p_root --key-file -".
 |> Of course i am no real Linux expert but only a do-it-yourself guy.

Well, and user space only, and never having read BIOS or UEFI
standards / implementations.
I have reread (almost) all of FreeBSDs documentation (loader, uefi
etc) today, and it is always great to see such good documentation!
Of course some things are beyond what a "normal" user would
understand, but at least there is documentation available.
That is really great.

 |> busybox allows me to manage this easily, to answer your question.
 |
 |You could do that on FreeBSD with a loader.efi that has a ram disk
 |built into it as well, including a 'beastie box' thing that's akin to
 |busybox.
 |It will boot in one step and no no further I/O to get a running system.
 |Others have used this for secure boot and to boot a small ram disk that's
 |later discarded as userland decides what root should be. But it's much less
 |automated than in Linux...

Hm, but wait.  Isn't it easier for FreeBSD?  I must admit i never
used FreeBSD/geli on bare metal, i started using fully encrypted
hard disks in mid 2021, no sooner.  So when i read [1] it seems
FreeBSD's boot procedure is so nicely linked with geli that the
boot loader asks itself for the password of the encrypted
partition, simply by setting some variables in loader.conf?
I mean, the /boot/ must be available?

  [1] https://man.freebsd.org/cgi/man.cgi?query=geli&apropos=0&sektion=0&manpath=FreeBSD+13.2-RELEASE+and+Ports&arch=default&format=html

These simple scripts i use for Linux also do not support true
suspend, only suspend-to-ram.  Ie, i have no idea how i could
"suspend" the encrypted device and then "resume" it again;
The "warm" security is only (initiated via ACPI event from root
account) auto-unload of SSH and PGP keys etc, unload of encfs fuse
volumes, and then slock of all X displays etc.  Ie, all programs
which run live on the encrypted volume.  When the laptop wakes up,
X is running etc.
One would need the possibility to wake up to "a password prompt".
Well, likely it would work to mount EFI vfat before the suspend,
create a console and auto-switch to it, and then sit there, ready
for reading in the password once the system resumes.  That is to
say: i always believed in FreeBSD this works just as easy as the
boot procedure?

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 5531 bytes --]

On Wed, Sep 6, 2023 at 6:12 PM Steffen Nurpmeso <steffen@sdaoden.eu> wrote:

> Hello!
>
> Warner Losh wrote in
>  <CANCZdfoy0UJyFdvJ_fO4FDks_dfEOhsJo+7p2RORdk6gQWf7qg@mail.gmail.com>:
>  |On Tue, Sep 5, 2023 at 9:53 AM Steffen Nurpmeso <steffen@sdaoden.eu>
> wrote:
>  |> Steffen Nurpmeso wrote in
>  |>  <20230904221059.sF2G0%steffen@sdaoden.eu>:
>  |>|Norman Wilson wrote in
>  |>| <9A989054DE79CE5059CBA74797391E39.for-standards-violators@oclsc.org>:
>  |>  ...
>  |>||Perhaps the question to ask is why such a magic program is
>  |>||needed at all.  Is it just because programs like the shell
>  |>||have become so large and unwieldy that they won't fit in
>  |>||a small environment suitable for loading into an initramfs?
>  |>  ...
>  |>|For my laptop it allows me easy boot management.
>  ...
>  |> Only to add that this is because of Linux and the way it is doing
>  |> things.  If i would use FreeBSD on bare metal, then i would have
>  |> an EFI boot loader on EFI that knows (only) enough to ask for
>  |> passphrase (correct me if i am wrong), and can then boot the
>  |> kernel from FFS or ZFS.  (You have to choose dedicated ZFS boot
>  |> loader iirc, but despite that...)
>  |
>  |No, you don't have to choose the dedicated ZFS boot loader, at least not
>  |anymore.
>
> Ah!  It seems regarding FreeBSD i am stuck in BIOS world, then.
> (zfsboot and gptzfsboot.)
>

Yes, for MBR, things are necessarily more complicated because the ZFS
boot loader is too big. The first few sectors live in the 8k space that the
traditional boot loader lives, and the rest is DD'd to a special area of
the ZFS
uberblock reserved for boot blocks. For gptboot vs gptzfsboot, they were
written to be either or and combining them presented challenges, not least
was
the partition size that gptboot lives in was created for many releases too
small
to hold gptzfsboot and making it tricky to upgrade...


>  |Also, you can use boot1.efi to load loader.efi from the root filesystem
> to
>
> The former deprecated i see.
>
>  |load the kernel, or you could use loader.efi directly on the ESP to load
>  |the kernel. boot1 barely knows anything (and has only one choice of
>  |what to boot). loader.efi is the full deal, and can do rather a lot of
>  |sophisticated things.
>
> Great.
>
>   ...
>  |> But anyhow.  With an EFI_STUB Linux kernel i can save me all that,
>  |> with busybox i get a complete environment (i then even create an
>  |> initrd in /boot/ on the fly so i do not have to type the password
>  |> a second time, that can (optionally) be cached, and is, actually
>  ...
>  |> Unfortunately cryptsetup is needed even though, i think, the
>  |> kernel has anything needed; you just cannot access it.  cryptsetup
>  |> is only needed for "$cs open $PART_ROOT p_root --key-file -".
>  |> Of course i am no real Linux expert but only a do-it-yourself guy.
>
> Well, and user space only, and never having read BIOS or UEFI
> standards / implementations.
> I have reread (almost) all of FreeBSDs documentation (loader, uefi
> etc) today, and it is always great to see such good documentation!
> Of course some things are beyond what a "normal" user would
> understand, but at least there is documentation available.
> That is really great.
>
>  |> busybox allows me to manage this easily, to answer your question.
>  |
>  |You could do that on FreeBSD with a loader.efi that has a ram disk
>  |built into it as well, including a 'beastie box' thing that's akin to
>  |busybox.
>  |It will boot in one step and no no further I/O to get a running system.
>  |Others have used this for secure boot and to boot a small ram disk that's
>  |later discarded as userland decides what root should be. But it's much
> less
>  |automated than in Linux...
>
> Hm, but wait.  Isn't it easier for FreeBSD?  I must admit i never
> used FreeBSD/geli on bare metal, i started using fully encrypted
> hard disks in mid 2021, no sooner.  So when i read [1] it seems
> FreeBSD's boot procedure is so nicely linked with geli that the
> boot loader asks itself for the password of the encrypted
> partition, simply by setting some variables in loader.conf?
> I mean, the /boot/ must be available?
>
>   [1]
> https://man.freebsd.org/cgi/man.cgi?query=geli&apropos=0&sektion=0&manpath=FreeBSD+13.2-RELEASE+and+Ports&arch=default&format=html
>
> These simple scripts i use for Linux also do not support true
> suspend, only suspend-to-ram.  Ie, i have no idea how i could
> "suspend" the encrypted device and then "resume" it again;
> The "warm" security is only (initiated via ACPI event from root
> account) auto-unload of SSH and PGP keys etc, unload of encfs fuse
> volumes, and then slock of all X displays etc.  Ie, all programs
> which run live on the encrypted volume.  When the laptop wakes up,
> X is running etc.
> One would need the possibility to wake up to "a password prompt".
> Well, likely it would work to mount EFI vfat before the suspend,
> create a console and auto-switch to it, and then sit there, ready
> for reading in the password once the system resumes.  That is to
> say: i always believed in FreeBSD this works just as easy as the
> boot procedure?
>

Yea. I find it easier :). But there's some better automation available in
Linux for the 'complicated' situations. I was also trying to be generous
to Linux because I know the passion of its supporters in the face of
even mild criticism.

Warner

[-- Attachment #2: Type: text/html, Size: 7144 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Theodore Ts'o]

On Thu, Sep 07, 2023 at 10:05:37AM -0600, Warner Losh wrote:
> 
> Yea. I find it easier :). But there's some better automation available in
> Linux for the 'complicated' situations.

Yeah, this is why most of the Linux distributions always use an
initramfs, at least by default, because it handles all (or at least
most) of these complicated sitautions automatically.  So this leads to
people assuming that Linux "requires" using a initramfs, when it would
be more accurate to say that most Linux distributions makes it super
easy to use an initramfs, and much more difficult (because the
documentation is a bit scattered, and many may be somewhat out of
date) to do things without an Linux distribution-engineered initramfs.

And the other clear difference between Linux and FreeBSD is that
things like initramfs tends to be distribution-specific, so there is
competition between distributions to see who can make a simpler/easier
installer and boot sequnce, and so market forces tends to strongly
encourage developers (who after all, tend to be experts and who don't
need the installer and boot up automation), to improve things much
more aggressively.  (And since there are product managers involved,
often features and time to market often trump issues like technical
debt.)

The downside with this approach, though, is that distributions tend to
reinvent the wheel multiple times, and the initramfs infrastructures
are different.  So learning how things work in say, Debian, isn't
going to help you understand the fine details of how Fedora or Red Hat
Enterprise Linux works except at a very basic, high level.  And the
same is most definitely true if you compare the installers between
Debian, Ubuntu, Fedora, etc.

FreeBSD has the advantage that it has a much more centralized
development model, especially as it relates to kernel/libc
integration, the installer, and the boot loader path.  But while there
is a certain amount of competitive forces of FreeBSD, NetBSD, OpenBSD,
et. al., improving and simplifying the boot loader, from an outsider
looking in, it would appear that those forces aren't quite as strong,
and the level of convergence beween the *BSD seems to be at least as
strong, if not stronger, in some areas, than between Linux
distributions.

Cheers,

					- Ted

[TUHS] Re: Unix install & "standalone" package

[Michael Kjörling]

On 5 Sep 2023 17:53 +0200, from steffen@sdaoden.eu (Steffen Nurpmeso):
> Unfortunately cryptsetup is needed even though, i think, the
> kernel has anything needed; you just cannot access it.  cryptsetup
> is only needed for "$cs open $PART_ROOT p_root --key-file -".
> Of course i am no real Linux expert but only a do-it-yourself guy.

If your need is restricted to a highly specific use case and you are
trying to keep it as small as possible, then it should be possible to
write a custom wrapper around whatever libcryptsetup functionality you
need and avoid the extra code that you get with cryptsetup proper.

-- 
Michael Kjörling                     🔗 https://michael.kjorling.se
“Remember when, on the Internet, nobody cared that you were a dog?”

[TUHS] Re: Unix install & "standalone" package

[Steffen Nurpmeso]

Michael Kjörling wrote in
 <f948d06c-14f4-40bd-8e32-4db1c5b1dd21@home.arpa>:
 |On 5 Sep 2023 17:53 +0200, from steffen@sdaoden.eu (Steffen Nurpmeso):
 |> Unfortunately cryptsetup is needed even though, i think, the
 |> kernel has anything needed; you just cannot access it.  cryptsetup
 |> is only needed for "$cs open $PART_ROOT p_root --key-file -".
 |> Of course i am no real Linux expert but only a do-it-yourself guy.
 |
 |If your need is restricted to a highly specific use case and you are
 |trying to keep it as small as possible, then it should be possible to
 |write a custom wrapper around whatever libcryptsetup functionality you
 |need and avoid the extra code that you get with cryptsetup proper.

It is nicely documented, and my Linux distribution ships the
static library anyhow.  But i am a lazy sort regarding such,
i just take the thing of my distribution and copy it over (they do
build it statically also by default).
You know, things change, and if you do not follow closely, you
stand in the rain.  I am not a paid Linux engineer that follows
this rapidly moving target in the end.
For example the (no longer) new random developer chose to disable
feeding entropy via /dev/urandom, here (distribution) still is

  # Load random seed
  /bin/cat /var/lib/urandom/seed > /dev/urandom

for almost two decades (it is a rather young one), but the code
path was mutilated (i read the kernel source once he had rewritten
that to be blake2/some 32-byte block thing based), now one needs
to use some ioctl interface fwiw.
Or once here cryptsetup was updated to use OpenSSL 3.0 suddenly
ripemd160 was no longer available on EFI (aka purely static,
without the filesystem avaialable), even though its release notes
explicitly mentioned the problem as solved, and OpenSSL 3's
libcrypto.a _had_ ripemd160...  I had to switch to sha512 .. then
to sha256 once cryptsetup started warning args had to be explicit
in the future.  Mind you, i in fact use it twice, also for
encrypted swap, i only wrongly searched for $cs

    $2 open --type plain --cipher aes --key-size 256 \
      --hash sha256 $PART_SWAP p_swap --key-file - &&

i said on IRC

  cryptsetup does EVP_DigestInit_ex(h->md, h->hash_id, NULL),
  i presume that does load additional things.

That surely is it, i did not track it further.
So no, to answer you, i have no highly specific use case at all.
This is only an encrypted volume with my own boot-style that
requires no boot loader but Linux itself.
Maybe i should really look deeply in how cryptsetup then attaches
a LUKS2 volume to the kernel, maybe it actually _would_ be
possible to do this simply in some other way.
But truly writing a program?  I feel much saver in the horde, with
so many people, specialists even, working on Linux, LUKS2,
cryptsetup, OpenSSL, .. these are all moving targets.
(I mean, i am lucky if i _can_ do a bit of programming on at least
the MUA i maintain; so much to do!  And roff hopefully somewhere
on the horizon, somewhen; today it was zero minutes.)

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: Unix install & "standalone" package

[Steffen Nurpmeso]

[-- Attachment #1: Type: text/plain, Size: 7275 bytes --]

Steffen Nurpmeso wrote in
 <20230908233854.Xni_j%steffen@sdaoden.eu>:
 |Michael Kjörling wrote in
 | <f948d06c-14f4-40bd-8e32-4db1c5b1dd21@home.arpa>:
 ||On 5 Sep 2023 17:53 +0200, from steffen@sdaoden.eu (Steffen Nurpmeso):
 ...
 ||> Of course i am no real Linux expert but only a do-it-yourself guy.
 ||
 ||If your need is restricted to a highly specific use case and you are
 ||trying to keep it as small as possible, then it should be possible to
 ...

Actually maybe someone may find it funny, even though it is
neither historic nor very sophisticated.  Let me post it.
In the end it takes some time to get there.


I would expect almost all of you will not be interested in the
lengthy rest of this mail.
Ciao, and a nice Sunday i wish from Germany!


All you need are statically linked busybox (and cryptsetup with
encryption), and a kernel with EFI_STUB on the EFI partition, so
it can be booted directly.  The distribution kernels i have
(ArchLinux, AlpineLinux) have it enabled; if your EFI is large
enough you could simply copy it and all its masses of modules to
EFI.

Otherwise the kernel needs all the firmware and modules to boot
and to access all filesystems where the real stuff is.  My own
kernel has a lot of things statically built-in, but it can load
more via modules (stage2 uses "the same" kernel later, and does).
Kernels on stage2 need kexec, i have

  CONFIG_KEXEC=y
  CONFIG_KEXEC_FILE=y
  CONFIG_ARCH_HAS_KEXEC_PURGATORY=y
  # CONFIG_KEXEC_SIG is not set
  CONFIG_KEXEC_CORE=y

-s1.sh and -s2.sh are docu-commented at the top; they can also be
used to create the necessary environment.  If i recall correctly

  cd /boot
  sh linit-init-s2.sh PATH-TO-BUSYBOX PATH-TO-CRYPTSETUP
  mount EFI-PARTITION
  cd EFI-PARTITION
  sh linit-init-s1.sh PATH-TO-BUSYBOX PATH-TO-CRYPTSETUP

This only creates files and directories etc, and beat me if it
does any harm otherwise.

I use a specific naming scheme, the script reacts in particular
on -old and -new to select sensible defaults.

  #?0|kent:/boot# ll ideapad*.efi
  -rwxr-xr-x 1 root root 10112672 Aug 26 18:44 ideapad-stage1.efi*
  -rw-r--r-- 1 root root 10120512 Sep  2 19:52 ideapad-6_1.efi
  -rw-r--r-- 1 root root 10120512 Sep  9 18:18 ideapad-6_1-old.efi
  -rw-r--r-- 1 root root 10121376 Sep  9 18:18 ideapad-6_1-new.efi

That very kernel (series) is in fact shared in between several
computers of a similar series, kent is the one i write this on:

  #?1|kent:/boot# v kent.sh
  #!/busybox.static sh
  #@ kent, step 1., via EFI.
  PART_ROOT=/dev/nvme0n1p?
  ROOT_DECRYPT='-t btrfs -o defaults,subvol=/crux/kent/root'
  	PART_ROOT1=/dev/nvme0n1p8
  	ROOT_DECRYPT1='-t btrfs -o defaults,subvol=/crux/kent/root.old'
  INIT_S2=/boot/kent-2.sh
  . /linux-init-s1.sh

  #?0|kent:/boot# v kent-2.sh
  #@ kent, stage 2
  KERNEL_ID=ideapad

^ restrict kernel selection for kexec:
^      	[ "${k##*$KERNEL_ID-*.efi}" != "$k" ] || continue

  PART_SWAP=/dev/nvme0n1p6
  SWAP_INPUT=y

^ you can have additional interactive random for swap encryption
^ (it is a lie i use that in practice)

  INITRD_PATH=/boot/.kent.initrd
  KEXEC_ARGS="--append=\"rtw88_pci.disable_aspm=1 rc.hostname=kent\""
  FILE_CHECK='kent-direct.sh kent.sh ideapad-stage1.efi'
  SWITCH_ROOT=media/initrd

Stage 2 config file is sourced automatically, so no need to source
linux-init-s2.sh yourself (stage 2 uses switch_root).
I also have a direct boot, but have not used it for long:

  #?0|kent:~# v /boot/kent-direct.sh
  #!/busybox.static sh
  #@ kent-direct, sole step 1., via EFI.
  PART_ROOT=/dev/nvme0n1p8
  ROOT_DECRYPT='-t btrfs -o defaults,subvol=/crux/kent/root'
  	PART_ROOT1=/dev/nvme0n1p8
  	ROOT_DECRYPT1='-t btrfs -o defaults,subvol=/crux/kent/root.old'
  PART_SWAP=/dev/nvme0n1p6
  PIVOT_OLDROOT=media/initrd
  . /linux-init-s1.sh

This does not use kexec, but only pivot_root's to the decrypted
filesystem, starting sbin/init there.  The kernel from EFI
continues to run.

It is pretty cool how i now can drive several computers easily
simply by creating a shell script with some variable assignments.
In fact the filesystem is shared in between multiple computers
(with only selective partitions being truly unique, but still
backed up everywhere), and all those text configs are in /boot.
Only the VFAT EFI partition is truly system-specific (ie, in that
it only holds the stage1 config file for $HOSTNAME).

I like that very much, and hope Linux continues to be so flexible.
(Not that one day you need systemd-early to get into Linux to get
to systemd-later, or something.)

The EFI boot is driven via efibootmgr(8) and has config like

  #@ /root/hosts/self/efiboot
  d=/dev/nvme0n1p1
  maxno=1

  b1=0x01  L1=kent  l1=ideapad-stage1
  u1='root='${d}' rootfstype=vfat init=/kent.sh'

driven by /root/bin/efiboot.sh (shortened):

  #!/bin/sh
  : ${HOSTNAME:=$(uname -n)}
  : ${DBG:=}
  if [ -f /root/hosts/${HOSTNAME}/efiboot ]; then
          . /root/hosts/${HOSTNAME}/efiboot
  else
          echo >&2 "MISS /root/hosts/${HOSTNAME}/efiboot"
          exit 1
  fi
  if [ "$1" = create ]; then
          obo=$(efibootmgr | grep -E ^BootOrder: | sed -E 's/^BootOrder:[[:space:]]*//')
          if [ -z "$obo" ]; then
                  echo >&2 'Cannot determine previous boot order'
                  exit 1
          fi
          nbo= i=1
          while [ $i -le $maxno ]; do
                  eval ${DBG} efibootmgr -c -b \$b$i -L \\\"\$L$i\\\" -d \$d$i -l \\\"\$l$i.efi\\\" -u \\\"\$u$i\\\" #>/dev/null
                  [ -n "$nbo" ] && nbo=$nbo,
                  eval nbo=\$nbo\$b$i
                  i=$((i + 1))
          done
          ${DBG} efibootmgr -o $nbo,$obo #> /dev/null
  elif [ "$1" = delete ]; then
          i=1
          while [ $i -le $maxno ]; do
                  eval ${DBG} efibootmgr -B -b \$b$i #>/dev/null
                  i=$((i + 1))
          done
  else
          echo >&2 'Synsopsis: efiboot.sh create|delete'
          exit 1
  fi

  ${DBG} efibootmgr -v -u

This is of course git managed, so i can distribute it easily.
(Further even, the entire ideapad series configurations differ in
only 193 lines, and most of those are the different SSH keys.
The rest are different device names, ie network, display, volume
etc, disk partitions, host names.)

I am pretty lucky i found a way to manage my systems like this.
I have several git branches in root.git, linux.kconfig, bin,
iwd.network, and one for each computer.  I can then "git merge"
selectively together what is necessary.  For example my web
vserver only needs bin and its own configuration, whereas the
laptops also need iwd.network.  Etc etc.  Like this every computer
only knows the selective minimum.  The full git repo is on an
encrypted volume stored away.
And it is not Puppet or Ansible or any of those programs, it is
nothing but git and its branches.
But now i really left manually driven cryptsetup by far.
Ciao!

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[-- Attachment #2: linux-init-s1.sh --]
[-- Type: application/x-sh, Size: 4596 bytes --]

#@ /linux-init-s1.sh, simple Linux init, stage 1.
#@ To be sourced from a kernel command line init=SCRIPT like:
#@
#@   #!/busybox.static sh
#@   #@ For kent
#@
#@   PART_ROOT=/dev/nvme0n1p8
#@   ROOT_DECRYPT='-t btrfs -o defaults,subvol=/crux/kent/root'
#@   PART_SWAP=/dev/nvme0n1p6
#@   INIT_S2=boot/kent-2.sh
#@
#@   . /linux-init-s1.sh
#@
#@ Alternatively call AS ROOT and FROM WITHIN directory that boots like
#@   sh linux-init-s1.sh PATH_TO_BUSYBOX PATH_TO_CRYPTSETUP
#@ and is expected to be mounted as / on system start, for init purposes.

# Non-empty and we run mdev -s very first (value passed to it via ARGV)
: ${MDEV_RUN:=}

# Unless ROOT_DECRYPT (only partition then) this can be mount spec, like
#		PART_ROOT='-t btrfs -o defaults,subvol=/crux/kent/root'
[ $# -eq 0 ] &&
	: ${PART_ROOT:?"Need a usable \$PART_ROOT"}
# Non-empty and we decrypt ROOT; _it_ should be mount spec then
: ${ROOT_DECRYPT:=}
# It is possible to specify multiple PART_ROOT[/ROOT_DECRYPT] tuples by
# giving PART_ROOT1, 2, 3...	A dialog for choosing one to use then appears

# Non-empty and we create encrypted swap from the given partition
: ${PART_SWAP:=}
# Non-empty and we ask for more random bytes for swap encryption
: ${SWAP_INPUT:=}

# Non-empty and we give up to the stage 2 -s2.sh that is expected to live
# in [/dev/mapper/p_root]/boot.	ONLY WITH $ROOT_DECRYPT!
# The value is the config file of stage 2 as an _absolute_ path within the
# newly mounted root.
: ${INIT_S2:=}

# Were we move old root to when doing pivot_root(8) in non-$INIT_S2 cases
: ${PIVOT_OLDROOT:=media/efi}

BB=busybox.static
CS=cryptsetup.static

# -- >8 -- 8< --

xroot() { # {{{
	bb=./$BB cs=./$CS
	slept= # May want to sleep to pass by kernel async console output

	xbase_mounts "$bb" # (First for completeness sake [not needed])

	#
	PART_SUFFIX=.0
	if [ -n "$PART_ROOT1" ]; then
		# Sleep a bit and hope kernel async console output passed by then
		[ -z "$slept" ] && $bb sleep 3
		slept=y

		while :; do
			echo
			echo '. Multiple $PART_ROOT[/$ROOT_DECRYPT] encountered:'
			echo "  [0] PART_ROOT=$PART_ROOT"
			[ -n "$ROOT_DECRYPT" ] && echo "  ROOT_DECRYPT=$ROOT_DECRYPT"
			ri=1
			while :; do
				eval rx="\$PART_ROOT$ri"
				[ -z "$rx" ] && break
				echo "  [$ri] ..ROOT$ri=$rx"
				eval rx="\$ROOT_DECRYPT$ri"
				[ -n "$rx" ] && echo "  ..CRYPT$ri=$rx"
				ri=$((ri + 1))
			done

			rmsg= rrt=
			if (echo y | read -t 1 chck) >/dev/null 2>&1; then
				rmsg=' (10 seconds until, or empty input for default)'
				rrt='-t 10'
			fi

			printf '. Please choose $PART_ROOT'"$rmsg"'? '
			read $rrt PART_SUFFIX
			if [ $? -ne 0 ] || [ -z "$PART_SUFFIX" ] || [ "$PART_SUFFIX" = 0 ]; then
				PART_SUFFIX=.0
				echo
				break
			elif [ "${PART_SUFFIX##*[^0-9]}" != "$PART_SUFFIX" ]; then
				echo >&2 '! Invalid input'
				continue
			fi

			eval ri=\$PART_ROOT$PART_SUFFIX
			if [ -z "$ri" ]; then
				echo >&2 '! No such PART_ROOT: '$PART_SUFFIX
			else
				PART_ROOT=$ri
				eval ROOT_DECRYPT=\$ROOT_DECRYPT$PART_SUFFIX
				PART_SUFFIX=.$PART_SUFFIX
				echo
				break
			fi
		done
	fi

	#
	if [ -n "$ROOT_DECRYPT" ]; then
		[ -z "$slept" ] && $bb sleep 3
		slept=y
		while :; do
			printf '. Please set free $PART_ROOT: '
			$bb stty -echo
			read ROOT_PASS
			$bb stty echo

			printf "%s" "$ROOT_PASS" | $cs open $PART_ROOT p_root --key-file - && break
			xconfirm 'Cannot cryptsetup open '"$PART_ROOT"', try again' || x 1
		done

		$bb mount $ROOT_DECRYPT /dev/mapper/p_root mnt || x 2

		if [ -n "$INIT_S2" ]; then
			echo '. Going $INIT_S2='"$INIT_S2"
			cd mnt || x 3

			boot/$BB mount --bind ../dev dev || x 10
			boot/$BB mount --bind ../sys sys || x 11
			boot/$BB mount --bind ../proc proc || x 12
			boot/$BB mount --bind ../run run || x 13

			exec boot/$BB chroot . /boot/$BB sh \
					/boot/linux-init-s2.sh --stage1 "$INIT_S2" \
						"$PART_SUFFIX" "$PART_ROOT" \
						"$ROOT_DECRYPT" "$ROOT_PASS" ||
				x 'Cannot go $INIT_S2'
		fi
	else
		echo '. Mounting $PART_ROOT'
		$bb mount $PART_ROOT mnt || x 2
	fi

	xswap $bb $cs

	echo '. About to pivot_root'
	cd mnt || x 3
	bb=boot/$bb
	$bb pivot_root . $PIVOT_OLDROOT || x 4

	$bb mount -o move $PIVOT_OLDROOT/dev dev || x 10
	$bb mount -o move $PIVOT_OLDROOT/sys sys || x 11
	$bb mount -o move $PIVOT_OLDROOT/proc proc || x 12
	$bb mount -o move $PIVOT_OLDROOT/run run || x 13

	echo '. Going sbin/init'
	$bb stty isig
	exec $bb chroot . sbin/init || x 'Cannot sbin/init'
} # }}}

if [ $# -ne 0 ]; then
	. $($1 dirname "$0")/linux-init-lib.sh
	xsetup "$1" "$2"
else
	. $(./$BB dirname "$0")/linux-init-lib.sh
	xroot
fi
exit 127

# s-sht-mode

[-- Attachment #3: linux-init-s2.sh --]
[-- Type: application/x-sh, Size: 7186 bytes --]

#@ /linux-init-s2.sh, simple Linux init, optional stage 2, called from stage 1.
#@ Requires $BB, $CS, as below, and these funs, on EFI / as well as new /boot/.
#@
#@   #@ For kent, stage 2
#@
#@   [KERNEL_ID=ideapad] - if set, only $KERNEL_ID-*.efi kernels are examined
#@   PART_ROOT=/dev/nvme0n1p8
#@   ROOT_DECRYPT='-t btrfs -o defaults,subvol=/crux-3.6/kent/root'
#@   PART_SWAP=/dev/nvme0n1p6
#@   INITRD_PATH=boot/.s2.initrd
#@   SWITCH_ROOT=media/initrd
#@
#@ Alternatively call AS ROOT and FROM WITHIN directory that boots like
#@   sh linux-init-s2.sh PATH_TO_BUSYBOX PATH_TO_CRYPTSETUP
#@ for init purposes.

# Identical to stage 1.
: ${MDEV_RUN:=}
#: ${PART_ROOT:=} via stage 1
#: ${ROOT_DECRYPT:=} via stage 1 (ditto ROOT_PASS)
: ${PART_SWAP:=}
: ${SWAP_INPUT:=}

# If non-empty, we try to load the given initrd (relative to root) instead of
# newly creating it; also, we will try to save it there if we had to create it
# Will get PART_SUFFIX of chosen PART_ROOT/ROOT_DECRYPT tuple!
: ${INITRD_PATH:=}

# Further arguments when kexec(8)ing the new kernel
: ${KEXEC_ARGS:=}
	: ${KEXEC_EXE:=usr/sbin/kexec}

# Running in the kexec(8) kernel, we may apply some file comparisons.
# If set it is a list of space separated files which must exist in /boot as
# well as $FILE_CHECK_PARTITION (most likely the same as $SWITCH_ROOT), that is
# temporarily mounted if it is not yet.
# The files $BB, $CS, linux-init-{lib,s1}.sh are implied
: ${FILE_CHECK:=}
	: ${FILE_CHECK_PARTITION:=media/efi}
	: ${FILE_CHECK_MOUNT_ARGS:=-n -t vfat /dev/nvme0n1p1}

# Where we move old root to when doing switch_root(8)
: ${SWITCH_ROOT:=media/initrd}

BB=busybox.static
CS=cryptsetup.static

# -- >8 -- 8< --

xkexec() { # $1=config file; assumes we are in / and boot/ "has it" {{{
	conf=$1

	# Find the possible kernels to boot
	set -- boot/*.efi
	ki=1 kdef= knew=
	for k
	do
		if [ -n "$KERNEL_ID" ]; then
			[ "${k##*$KERNEL_ID-*.efi}" != "$k" ] || continue
		fi
		if [ "${k##*-new.efi}" = "$k" ] && [ "${k##*-old.efi}" = "$k" ] && [ "${k##*-stage1.efi}" = "$k" ]; then
			#[ -z "$kdef" ] && kdef=$k
			kdef=$k
		fi
		if [ -z "$knew" ] && [ "${k##*-new.efi}" != "$k" ]; then
			knew=$k
		fi
		eval k$ki=$k
		ki=$((ki + 1))
	done

	# Let user choose one
	trap 'x "Got signal"' INT QUIT TERM
	echo '+ trapping INT/QUIT/TERM to start a shell'

	k= k0=-
	while [ -z "$k" ]; do
		echo
		echo '. Available kernels:'
		echo '  [0] Edit $KEXEC_ARGS'
		kj=1
		while [ $kj -lt $ki ]; do
			eval echo "\"	[$kj] \$k$kj\""
			kj=$((kj + 1))
		done
		echo

		if [ -n "$knew" ] && xconfirm 'Use the new kernel ('"$knew"')?'; then
			k=$knew
		else
			kmsg= krt=
			if [ -n "$kdef" ] && (echo y | read -t 1 chck) >/dev/null 2>&1; then
				kmsg=' (10 seconds until, or empty input for '"$kdef"')'
				krt='-t 10'
			fi

			printf '. Please choose kernel'"$kmsg"'? '
			read $krt kx
			if [ $? -ne 0 ] || [ -z "$kx" ]; then
				echo
				k=$kdef
			elif [ "${kx##*[^0-9]}" != "$kx" ]; then
				echo >&2 '! Invalid input'
				continue
			fi

			eval k=\$k$kx
			if [ -z "$k" ]; then
				echo >&2 '! No such kernel'
			elif [ "$k" = - ]; then
				printf "%s\n" "$KEXEC_ARGS" > run/.kexec
				echo '# Only first line counts.	Care for quoting!  :wq' run/.kexec
				boot/$BB vi run/.kexec
				k=$KEXEC_ARGS
				read KEXEC_ARGS < run/.kexec
				boot/$BB rm -f run/.kexec
				xconfirm "New KEXEC_ARGS=$KEXEC_ARGS" || KEXEC_ARGS=$k
				k=
			fi
		fi
	done

	# Create or reuse a temporary initrd that contains the password
	if [ -n "$INITRD_PATH" ] && [ -f "$INITRD_PATH$PART_SUFFIX" ] &&
			boot/$BB cp "$INITRD_PATH$PART_SUFFIX" run/.initrd; then
		echo '. Reusing initrd '"$INITRD_PATH$PART_SUFFIX"
	else
		echo '. Creating initrd'

		boot/$BB mount -o remount,exec run

		(
			set -e
			boot/$BB mkdir run/x
			cd run/x

			( xsetup2 "../../boot/$BB" "../../boot/$CS" )
			e=$?
			[ $e -ne 0 ] && exit $e

			./$BB mknod dev/console c 5 1 # redundant

			echo "#!/$BB sh" > ./init
			./$BB cat "$conf" >> ./init
			echo "PART_ROOT='$PART_ROOT'" >> ./init
			echo "ROOT_DECRYPT='$ROOT_DECRYPT'" >> ./init
			echo "ROOT_PASS='$ROOT_PASS'" >> ./init
			echo '. /linux-init-s2.sh' >> ./init
			./$BB chmod 0755 ./init

			{
				# Microcode update must be uncompressed and first
				[ -f ../../boot/early-ucode.cpio ] && ./$BB cat ../../boot/early-ucode.cpio
				# Followed by (possibly compressed) normal initrd
				./$BB find . | ./$BB cpio -H newc -o | ./$BB gzip -9 -n
			} > ../.initrd
		) || x 'Failed to create run/.initrd'

		boot/$BB mount -o remount,noexec run

		if [ -n "$INITRD_PATH" ]; then
			echo '. Saving initrd to '"$INITRD_PATH$PART_SUFFIX"
			boot/$BB cp run/.initrd "$INITRD_PATH$PART_SUFFIX"
			boot/$BB chmod 0600 "$INITRD_PATH$PART_SUFFIX"
		fi
	fi

	KEXEC_ARGS="$KEXEC_ARGS --initrd=run/.initrd"

	echo ". $KEXEC_EXE -l $k $KEXEC_ARGS"
	eval $KEXEC_EXE -l $k $KEXEC_ARGS || x 'Cannot invoke '"$KEXEC_EXE"' -l'
	echo ". $KEXE_EXE -e"
	exec $KEXEC_EXE -e || x 'Cannot invoke '"$KEXEC_EXE"' -e'
} # }}}

xfilecheck() { # $1=$BB; assumes we are in /mnt (_next_ root) {{{
	[ -z "$FILE_CHECK" ] && return
	echo '. Checking file equalities'

	umountpart=
	if xhavemnt "$1" "$FILE_CHECK_PARTITION"; then :; else
		umountpart=y
		$1 mount $FILE_CHECK_MOUNT_ARGS "$FILE_CHECK_PARTITION" || x 'Cannot mount '"$FILE_CHECK_PARTITION"
	fi

	err=
	$1 cmp -s "$1" "$FILE_CHECK_PARTITION"/$BB || err="$BB"
	$1 cmp -s boot/$CS "$FILE_CHECK_PARTITION"/$CS || err="$err $CS"
	$1 cmp -s boot/linux-init-lib.sh "$FILE_CHECK_PARTITION"/linux-init-lib.sh || err="$err init-lib.sh"
	$1 cmp -s boot/linux-init-s1.sh "$FILE_CHECK_PARTITION"/linux-init-s1.sh || err="$err init-s1.sh"

	if [ -f boot/early-ucode.cpio ] && [ -f "$FILE_CHECK_PARTITION"/early-ucode.cpio ]; then
		$1 cmp -s boot/early-ucode.cpio "$FILE_CHECK_PARTITION"/early-ucode.cpio || err="$err early-ucode.cpio"
	fi

	for f in $FILE_CHECK; do
		if [ -f "boot/$f" ] && [ -f "$FILE_CHECK_PARTITION/$f" ]; then
			$1 cmp -s "boot/$f" "$FILE_CHECK_PARTITION/$f" || err="$err $f"
		else
			err="$err (MISS: $f)"
		fi
	done

	[ -n "$umountpart" ] && $1 umount "$FILE_CHECK_PARTITION"

	if [ -z "$err" ]; then
		echo '  . Files successfully checked'
	else
		echo >&2 '! BOOT: $FILE_CHECK errors: '$err
		echo '! BOOT: $FILE_CHECK errors: '$err >> etc/motd
		$1 sleep 10
	fi
} # }}}

xroot() { # {{{
	# On initrd rootfs
	bb=./$BB cs=./$CS

	xbase_mounts "$bb"

	echo '. Setting free $PART_ROOT'
	printf "%s" "$ROOT_PASS" | $cs open $PART_ROOT p_root --key-file - || x 'Cannot cryptsetup open '"$PART_ROOT"

	$bb mount $ROOT_DECRYPT /dev/mapper/p_root mnt || x 2

	bb=mnt/boot/$BB
	cs=mnt/boot/$CS

	xswap $bb $cs

	cd mnt || x 3
	bb=boot/$BB
	cs=boot/$CS

	xfilecheck $bb

	echo '. Going switch_root sbin/init'
	$bb stty isig
	# -c dev/console
	exec $bb switch_root . sbin/init 2>&1 || x 'Cannot switch_root'
} # }}}

if [ "$1" = --stage1 ]; then
	cnf=$2
	PART_SUFFIX=$3
	PART_ROOT=$4
	ROOT_DECRYPT=$5
	ROOT_PASS=$6

	[ -f "$cnf" ] || x 'Cannot find $INIT_S2: '"$cnf"
	. "$cnf"
	. /boot/linux-init-lib.sh
	xkexec "$cnf"
elif [ $# -eq 2 ]; then
	. $($1 dirname "$0")/linux-init-lib.sh
	xsetup2 "$1" "$2" -
else
	. /linux-init-lib.sh
	xroot
fi
exit 127

# s-sht-mode

[-- Attachment #4: linux-init-lib.sh --]
[-- Type: application/x-sh, Size: 3646 bytes --]

#@ linux-init-lib.sh, helpers for simple init stage 1 and 2; will be sourced.

# misc {{{
x() {
	echo >&2 'ERROR ('"$*"')! Going shell'
	[ -x ./$BB ] && exec ./$BB sh
	[ -x /$BB ] && exec /$BB sh
	[ -x /boot/$BB ] && exec /boot/$BB sh
	[ -x /bin/sh ] && exec /bin/sh
	exit 100
}

xconfirm() {
	printf -- '. '"$*"' [Yy/else]? '
	read answer
	case $answer in
	[Yy]*) return 0;;
	*) return 1;;
	esac
}

_mpx= _mp=
xhavemnt() { # $1=$BB $2=file
	if [ "$2" = dev ]; then
		[ -e /dev/console ] && [ -e /dev/null ] && [ -e /dev/zero ]
		return $?
	elif [ "$2" = proc ]; then
		[ -e /proc/1 ] && [ -e /proc/self ] && [ -e /proc/uptime ]
		return $?
	elif [ "$2" = sys ]; then
		[ -e /sys/kernel ] && [ -e /sys/block ] && [ -e /sys/fs ]
		return $?
	fi

	if [ -z "$_mpx" ]; then
		_mpx=y
		$1 mountpoint -q / >/dev/null 2>&1 && _mp=y
	fi

	if [ -n "$_mp" ]; then
		$1 mountpoint -q "$2" >/dev/null 2>&1
		return $?
	elif [ -f /proc/mounts ]; then
		$1 grep -q ' '"$2"' ' /proc/mounts >/dev/null 2>&1
		return $?
	fi
	return 1
}

_xrand=
xrand() {
	_xrand=$(( (($RANDOM << 20) | ($RANDOM << 10) | ($RANDOM % 0x3FF)) ^ \
			(($RANDOM << 20) | ($RANDOM << 10) | ($RANDOM % 0x3FF))))
}
# }}}

xbase_mounts() { # {{{ $1=$BB [uses $MDEV_RUN; calls stty -isig]
	echo '. Checking /{dev,sys,proc,run}'

	xhavemnt $1 dev || $1 mount -n -t devtmpfs -o mode=0755,exec,nosuid,noatime none dev
	xhavemnt $1 sys || $1 mount -n -t sysfs none sys
	xhavemnt $1 proc || $1 mount -n -t proc none proc
	xhavemnt $1 run || $1 mount -n -t tmpfs -o mode=0755,noexec,nosuid,nodev none run

	#[ -e /sys/firmware/efi ] &&
	#mount -t efivarfs efivarfs /sys/firmware/efi/efivars -o nosuid,nodev,noexec

	$1 stty -isig

	if [ -n "$MDEV_RUN" ]; then
		echo '. About to run mdev -s, waiting for some more settlement'
		$1 sleep 8
		$1 mdev -s $MDEV_RUN
	fi
} # }}}

xsetup_base() { # $1=$BB $2=$CS [$3=no mdev stuff if set] {{{
	$1 cp -f "$1" ./$BB
	./$BB chmod 0700 ./$BB

	./$BB cp -f "$2" ./$CS
	./$BB chmod 0700 ./$CS

	./$BB cp -f "$0" .
	./$BB cp -f $(./$BB dirname "$0")/linux-init-lib.sh ./linux-init-lib.sh

	if [ $# -ne 3 ]; then
		# Prepare for $MDEV_RUN, too!
		./$BB mkdir etc bin dev mnt proc run sys $mdir

		./$BB cp ./$BB bin/sh
		./$BB chmod 0700 bin/sh

		echo ".* 0:0 660 @/$BB sh /etc/mdev.sh" > etc/mdev.conf
		./$BB cat > etc/mdev.sh <<-_EOT
			#@ /etc/mdev.sh for ./mdev.conf
			
			run() {
				if [ "\$ACTION" = add ]; then
					case "\$SUBSYSTEM" in
					block) block;;
					*) ;;
					esac
				fi
			
				exit
			}
			
			block() {
				[ -n "\$MDEV" ] || return
			
				[ -d /dev/disk ] || /$BB mkdir -p /dev/disk/by-uuid
			
				uuid=\$(/$BB blkid "\$MDEV" \| /$BB sed -Ee "s/.*UUID=\"([^\"]+)\".*/\1/")
				[ -n "\$uuid" ] || return
			
				/$BB ln -s ../../"\$MDEV" /dev/disk/by-uuid/"\$uuid"
			}
			
			run
		_EOT

		./$BB chmod 0700 etc/mdev.sh
	fi

	echo '! Plus kernel, init=SCRIPT, to be placed here'
}

xsetup() {
	xsetup_base "$@"
	exit
}

xsetup2() {
	xsetup_base "$@"
	# For $FILE_CHECK
	./$BB cp -f $(./$BB dirname "$0")/linux-init-s1.sh ./linux-init-s1.sh
	exit
} # }}}

xswap() { # $1=$BB $2=$CS {{{
	[ -z "$PART_SWAP" ] && return
	echo '. Encrypted swap'

	rand=
	if [ -n "$SWAP_INPUT" ]; then
		echo '  ... please type a line of randoms'
		$1 stty -echo >/dev/null 2>&1
		read rand
		$1 stty echo >/dev/null 2>&1
	else
		xrand
		rand=$_xrand
	fi

	xrand
	xr1=$_xrand
	xrand
	xr2=$_xrand
	xrand
	xr3=$_xrand
	xrand
	xr4=$_xrand
	xrand
	xr5=$_xrand

	echo "$xr1$rand$xr2$xr3$rand$xr4$xr5" |
			$2 open --type plain --cipher aes --key-size 256 --hash sha256 $PART_SWAP p_swap --key-file - &&
		$1 mkswap /dev/mapper/p_swap
} # }}}

# s-sht-mode

[TUHS] Re: Unix install & "standalone" package

[Theodore Ts'o]

On Sat, Sep 09, 2023 at 01:38:54AM +0200, Steffen Nurpmeso wrote:
> You know, things change, and if you do not follow closely, you
> stand in the rain.  I am not a paid Linux engineer that follows
> this rapidly moving target in the end.
> For example the (no longer) new random developer chose to disable
> feeding entropy via /dev/urandom, here (distribution) still is
> 
>   # Load random seed
>   /bin/cat /var/lib/urandom/seed > /dev/urandom
> 
> for almost two decades (it is a rather young one), but the code
> path was mutilated (i read the kernel source once he had rewritten
> that to be blake2/some 32-byte block thing based), now one needs
> to use some ioctl interface fwiw.

Huh?  That's not correct.  You can still introduce entropy into the
random pool by writing to /dev/urandom or /dev/random.  It is true
that there has been some changes to the design of /dev/random, but I
assure you as the original /dev/random developer, was consulted and
involved in the design discussions.

Anyway, this is off-topic for TUHS, but if you have specific
questions/complaints, feel free to address them to me and Jason.  Your
comments do make me suspect that there are some fundamental
misunderstandings at work.

Cheers,

					- Ted

[TUHS] Re: Unix install & "standalone" package

[Steffen Nurpmeso]

Theodore Ts'o wrote in
 <20230911041059.GH701295@mit.edu>:
 |On Sat, Sep 09, 2023 at 01:38:54AM +0200, Steffen Nurpmeso wrote:
 |> You know, things change, and if you do not follow closely, you
 |> stand in the rain.  I am not a paid Linux engineer that follows
 |> this rapidly moving target in the end.
 |> For example the (no longer) new random developer chose to disable
 |> feeding entropy via /dev/urandom, here (distribution) still is
 |> 
 |>   # Load random seed
 |>   /bin/cat /var/lib/urandom/seed > /dev/urandom
 |> 
 |> for almost two decades (it is a rather young one), but the code
 |> path was mutilated (i read the kernel source once he had rewritten
 |> that to be blake2/some 32-byte block thing based), now one needs
 |> to use some ioctl interface fwiw.
 |
 |Huh?  That's not correct.  You can still introduce entropy into the
 |random pool by writing to /dev/urandom or /dev/random.  It is true
 |that there has been some changes to the design of /dev/random, but I
 |assure you as the original /dev/random developer, was consulted and
 |involved in the design discussions.

Ah, lesser and lesser freedom for man!  That is the truth!
Neither was (last i looked) that counted no more to unlock man
from the chokehold of missing entropy, nor do newer kernels then
decrement entropy when you read random.

 |Anyway, this is off-topic for TUHS, but if you have specific
 |questions/complaints, feel free to address them to me and Jason.  Your
 |comments do make me suspect that there are some fundamental
 |misunderstandings at work.

Surely for the latter.  So much i give in on this song line.

Greetings.

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: Unix install & "standalone" package

[Jonathan Gray]

On Mon, Sep 04, 2023 at 10:44:21AM -0400, Norman Wilson wrote:
> I don't remember any special many-programs-in-one binary
> like busybox in any Unix from the days when Unix was simple
> enough for me to understand.  That covers the entire lifetime
> of the Research systems, but also System V and the BSDs and
> their sundry offspring up into at least the 1990s.
> 
> I'm pretty sure OpenBSD at least still has nothing like
> busybox.  The nearest thing was to make sure certain programs

look at ls -i output on the install media/bsd.rd
https://man.openbsd.org/crunchgen.8
Added to FreeBSD and NetBSD in 1994.

[TUHS] Unix install & "standalone" package

[Paul Ruizendaal via TUHS]

Recently, I was looking into the “Das U-Boot” boot loader package. Summarised with great simplification, u-boot bundles device drivers, file systems, commands and a Bourne-like shell into a standalone package. Normally it auto-runs a script that brings up a system, but when used in interactive mode it allows a great deal of poking around.

It made me think of the “standalone” set of programs for installing early Unix. On 16-bit understandably each basic command has to be a separate standalone program, but after the shift to 32-bit bundling more functionality in a single binary would have become possible.

How did the Unix “standalone” package evolve in the 80’s, both in the research and BSD lineages? Is there any retrospective paper about that? Or is it a case of “Use the source, Luke”?

[TUHS] Re: Unix install & "standalone" package

[emanuel stiebler]

On 2023-09-04 05:57, Paul Ruizendaal via TUHS wrote:
> 
> Recently, I was looking into the “Das U-Boot” boot loader package. Summarised with great simplification, u-boot bundles device drivers, file systems, commands and a Bourne-like shell into a standalone package. Normally it auto-runs a script that brings up a system, but when used in interactive mode it allows a great deal of poking around.

U-Boot has his roots in embedded systems, where you have a nice 
environment to debug your system. You have already everything there to 
download files with s-record from serial, boot from tftp, etc.

Actually also setting up the memory controllers, timing, setting 
environment variables ...

This became so comfortable, that sometimes the U-Boot code is larger 
than the target system. So, after development is done, the system jumps 
into the real application or code ...

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 7235 bytes --]

On Mon, Sep 4, 2023 at 3:58 AM Paul Ruizendaal via TUHS <tuhs@tuhs.org>
wrote:

>
> Recently, I was looking into the “Das U-Boot” boot loader package.
> Summarised with great simplification, u-boot bundles device drivers, file
> systems, commands and a Bourne-like shell into a standalone package.
> Normally it auto-runs a script that brings up a system, but when used in
> interactive mode it allows a great deal of poking around.
>
> It made me think of the “standalone” set of programs for installing early
> Unix. On 16-bit understandably each basic command has to be a separate
> standalone program, but after the shift to 32-bit bundling more
> functionality in a single binary would have become possible.
>
> How did the Unix “standalone” package evolve in the 80’s, both in the
> research and BSD lineages? Is there any retrospective paper about that? Or
> is it a case of “Use the source, Luke”?
>

The stand package continued in research and BSD to be those programs needed
to install and/or recover
badly damaged systems.  You could create a new file system, copy a file
from the tape to a partition, etc.
You couldn't do general scripting with this, by and large.

Originally, they were tape programs. This made sense because of its
original focus. In time, some systems
could load the stand alone programs instead of the kernel, but they
continued the original focus.

This is, imho, due in large part due to the miniroot. The miniroot evolved
into both a full-enough system
to do the installation scripts in shell instead of C (Venix, at least, had
their install program written in C).
You'd copy the minroot to swap and then install the system. But a number of
additional programs were
placed into the miniroot so you could do some limited filesystem repair,
file editing, etc.

In addition, many vendor's ROMs grew in complexity. Solbourne's ROMs, for
example, could do basic
repair of UFS (clri level, not fsck level), and copy files from one place
to another. I often recovered a
Solbourne system I screwed up by attaching an external SCSI drive that had
a known good kernel,
init, etc.

The 'stand' environment was a whole set of tools that could be used to
build stand-alone programs that
shared much code of their full unix brethren, despite not having a full
kernel under them. Kernel services
were provided by different libraries that did filesystem things, block
driver things, network things, etc
in a similar way to Unix, but with a much reduced footprint.

initramfs, as has been mentioned elsewhere, is pretty much a Linux
invention. It was designed to
'punt' on the choose where to load things from and have a very minimal
interface between the boot
loader and the system. In time, it grew to support more interfaces, more
ways of loading, and better
ways to mount something that you could then 'pivot' onto. Few other unix
systems went this route, though
many adopted some variation on the pivot_root functionality. Linux has
moved beyond the pivot root after
having booted the correct kernel into being able to take over the machine
early in, say, UEFI startup with
a minimal kernel and initramfs that just knows how to load the next kernel.
They skipped the complex boot
loader stage, and went straight to the 'run linux earlier' stage which is
how things like LinuxBoot, coreboot
and others have put the boot logic into bash scripts. The ability to
'kexec' a kernel and replace the current
running kernel originated in the 'non-stop' world that wanted to reduce
downtime. Now, it's used to reduce
firmware complexity by eliminating large swaths of UEFI from the boot
process, but also generalizes in
the embedded space.

FreeBSD, from around FreeBSD 2 (1995 or so), had /rescue which largely took
over form the stand alone environment
for the repair duties of things. FreeBSD also adopted a more complex boot
loader that would load the
kernel, modules, set tunables, etc prior to kicking off the kernel. Between
/rescue having all the tools needed
to repair bad updates, repair failing disks, get that one last backup
before the drive is dead while you wait
for the new drive to be delivered, etc, and /boot/loader being able to
script loading the kernel while the BIOS
was still around so the need for drivers in the loader was lessened.
However, as the BIOS evolved into UEFI
and FreeBSD pushed into the embedded space whose firmware provided a less
rich environment to the boot
loader, so it was able to load things off fewer and fewer devices, it
became clear that it would need a pivot
root feature to allow it to boot all the way into FreeBSD, load some
drivers from an included ram disk, and then
use that mount a new root and then 'reroot' to that by killing everything
and running init from that new root.
FreeBSD also moved from Forth to Lua in its scripting language for the boot
loader, giving 'pre boot'
environment support better features. I also added the ability to use
FreeBSD boot loader as a Linux binary
to load FreeBSD and its metadata from a LinuxBoot environment. Finally,
FreeBSD has 'spun out' and
generalized the /rescue feature to allow creation of any 'BeastyBox'
environment, similar to what you get
in a busy box, or clone, environment. This environment, though, is meant in
large part on both Linux and
FreeBSD to be in constrained environments where a full install is
prohibitive (even those that never pivot
to something more, like ap, routers and nas boxes).

NetBSD retains many of the old BSD stand-alone programs that started on the
vax. I've not studied things
beyond noticing this. OpenBSD is similar. Their boot chain is a bit simpler
than FreeBSD's, though there's
noises about porting FreeBSD's boot there. There's a port of /boot/loader
to illumos too, but I don't know
if it is the default, or just available. So I'll not chat about it more.

So the original 'standalone' environment where you had one program running
on a system has evolved
into either a rich boot loader environment that lets one do a lot to decide
what kernel to load, or towards
having a minimal selection of unix programs faster and using /bin/sh or
similar to do scripting. These
reduced environments are often called standalone, though all they share
just the name with the earlier
'stand' programs: they are full unix programs, but with reduced feature
sets and 'linker magic' to package
them in a way that's faster, smaller, etc (eg all in one binary). FreeBSD's
boot loader is an outgrowth
of the original standalone env, by way of a port of NetBSD's libsa.

I suspect in the future, we'll see more and more of a trend for low-level
init and then handing off to some
built-in kernel (be it Linux, BSD-based (there's now kexec), or whatever)
to reuse more of the vetted code
rather than re-inventing Unix inside the boot loader (which is a valid
criticism of FreeBSD's boot loader,
though it's rich feature set is what you get for the complexity).

Does that answer the prompt? Should I try to make this into more of a
retrospective paper and actually
do the research on the areas I was hand-wavy about?

Warner

[-- Attachment #2: Type: text/html, Size: 8399 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Dan Cross]

On Mon, Sep 4, 2023 at 1:08 PM Warner Losh <imp@bsdimp.com> wrote:
> [snip]
> There's a port of /boot/loader to illumos too, but I don't know
> if it is the default, or just available. So I'll not chat about it more.

I can confirm that it is the default on i86pc, though not universally.
At Oxide, for example, we boot directly into a very small loader held
in flash that has a compressed cpio archive containing the kernel and
a few necessary kernel modules compiled into it (as a byte blob); we
uncompress that blob into physical memory, locate and load the kernel
like a normal ELF binary, and jump to the kernel's ELF entry point,
passing a few basic arguments (notably, the location and length of the
cpio archive in physical memory). Entry to the kernel is thus in
64-bit mode with paging enabled.

> So the original 'standalone' environment where you had one program running on a system has evolved
> into either a rich boot loader environment that lets one do a lot to decide what kernel to load, or towards
> having a minimal selection of unix programs faster and using /bin/sh or similar to do scripting. These
> reduced environments are often called standalone, though all they share just the name with the earlier
> 'stand' programs: they are full unix programs, but with reduced feature sets and 'linker magic' to package
> them in a way that's faster, smaller, etc (eg all in one binary). FreeBSD's boot loader is an outgrowth
> of the original standalone env, by way of a port of NetBSD's libsa.
>
> I suspect in the future, we'll see more and more of a trend for low-level init and then handing off to some
> built-in kernel (be it Linux, BSD-based (there's now kexec), or whatever) to reuse more of the vetted code
> rather than re-inventing Unix inside the boot loader (which is a valid criticism of FreeBSD's boot loader,
> though it's rich feature set is what you get for the complexity).
>
> Does that answer the prompt? Should I try to make this into more of a retrospective paper and actually
> do the research on the areas I was hand-wavy about?

That would be interesting.

I can still remember booting IBM 6150 RTs into a miniroot environment
and using that to create and initialize filesystems when installing
AOS (4.3BSD for the RT) back in the day. To my mind, the standalone
programs were always oriented towards solving the related problems of
bootstrap initialization onto a fresh machine, and disaster recovery
when things were really, really messed up.

As I recall, the RT miniroot could either load from tape, or one could
`dd` it into swap and boot from that. In either case, I seem to recall
it was copied into memory and run as a RAM disk. The idea of busy-box
like "a bunch of utilities compiled into the same binary" was to save
space, particularly since this would be copied into RAM; even with
demand paging, redundant copies of bits from `libc` in each binary
were a waste for what was intended to be a minimal environment anyway.

        - Dan C.

[TUHS] Re: Unix install & "standalone" package

[Paul Ruizendaal via TUHS]

> Does that answer the prompt? Should I try to make this into more of a retrospective paper and actually
> do the research on the areas I was hand-wavy about?

That certainly answered the prompt, much appreciated the walkthrough. Currently just trying to get a view of the field and to collect recollections on how it was done back in the day.

Part of it is finding a conceptual framework that can make sense of it all.

One pitfall I would like to avoid in my own thinking is conflating “installing” and “booting”, even though the two seem related (one loads bits into permanent storage, the other into volatile storage; both are built around incrementally adding capability). When variable hardware comes into play, the two mix even more. Also related is the topic of recovery.

The starting point seems to be a setup where a small set of standalone programs is used to load or repair the bits, as was done for 16-bit unix.

A next conceptual step seems to be where first a very basic system is installed that is then used for further installation or for repair. This step seems to have come early, if this 32V install page is reflective of how it was done back in 1980: https://gunkies.org/wiki/Installing_32V_on_SIMH  The idea to use disk swap space for this also seems to have come early (and I suppose the concept lives on in “rescue partitions”).

Another conceptual step might be where early installer phases run a different, smaller kernel (or even OS) than the one being installed. There seems to be much potential for a “wheel of reincarnation” here where as an installer grows large, a pre-installer is created to load the installer and then the pre-installer grows large, etc. For booting, this wheel seems to have turned about 5 times in current Linux. Installing that from scratch on a fully blank SBC (without prepping a removable disk on another computer) also appears to have 4 or 5 revolutions. That is one revolution every 5 years for the 25 years from the late seventies to the early 2000’s.

Then there is the question of where in the "installer stack" to stop. My current interest excludes (precursors to) package managers, containers, etc.

In short: much to ponder, thanks to all for sharing recollections.


> On 4 Sep 2023, at 19:07, Warner Losh <imp@bsdimp.com> wrote:
> 
> 
> 
> On Mon, Sep 4, 2023 at 3:58 AM Paul Ruizendaal via TUHS <tuhs@tuhs.org> wrote:
> 
> Recently, I was looking into the “Das U-Boot” boot loader package. Summarised with great simplification, u-boot bundles device drivers, file systems, commands and a Bourne-like shell into a standalone package. Normally it auto-runs a script that brings up a system, but when used in interactive mode it allows a great deal of poking around.
> 
> It made me think of the “standalone” set of programs for installing early Unix. On 16-bit understandably each basic command has to be a separate standalone program, but after the shift to 32-bit bundling more functionality in a single binary would have become possible.
> 
> How did the Unix “standalone” package evolve in the 80’s, both in the research and BSD lineages? Is there any retrospective paper about that? Or is it a case of “Use the source, Luke”?
> 
> The stand package continued in research and BSD to be those programs needed to install and/or recover
> badly damaged systems.  You could create a new file system, copy a file from the tape to a partition, etc.
> You couldn't do general scripting with this, by and large.
> 
> Originally, they were tape programs. This made sense because of its original focus. In time, some systems
> could load the stand alone programs instead of the kernel, but they continued the original focus.
> 
> This is, imho, due in large part due to the miniroot. The miniroot evolved into both a full-enough system
> to do the installation scripts in shell instead of C (Venix, at least, had their install program written in C).
> You'd copy the minroot to swap and then install the system. But a number of additional programs were
> placed into the miniroot so you could do some limited filesystem repair, file editing, etc.
> 
> In addition, many vendor's ROMs grew in complexity. Solbourne's ROMs, for example, could do basic
> repair of UFS (clri level, not fsck level), and copy files from one place to another. I often recovered a
> Solbourne system I screwed up by attaching an external SCSI drive that had a known good kernel,
> init, etc.
> 
> The 'stand' environment was a whole set of tools that could be used to build stand-alone programs that
> shared much code of their full unix brethren, despite not having a full kernel under them. Kernel services
> were provided by different libraries that did filesystem things, block driver things, network things, etc
> in a similar way to Unix, but with a much reduced footprint.
> 
> initramfs, as has been mentioned elsewhere, is pretty much a Linux invention. It was designed to
> 'punt' on the choose where to load things from and have a very minimal interface between the boot
> loader and the system. In time, it grew to support more interfaces, more ways of loading, and better
> ways to mount something that you could then 'pivot' onto. Few other unix systems went this route, though
> many adopted some variation on the pivot_root functionality. Linux has moved beyond the pivot root after
> having booted the correct kernel into being able to take over the machine early in, say, UEFI startup with
> a minimal kernel and initramfs that just knows how to load the next kernel. They skipped the complex boot
> loader stage, and went straight to the 'run linux earlier' stage which is how things like LinuxBoot, coreboot
> and others have put the boot logic into bash scripts. The ability to 'kexec' a kernel and replace the current
> running kernel originated in the 'non-stop' world that wanted to reduce downtime. Now, it's used to reduce
> firmware complexity by eliminating large swaths of UEFI from the boot process, but also generalizes in
> the embedded space.
> 
> FreeBSD, from around FreeBSD 2 (1995 or so), had /rescue which largely took over form the stand alone environment
> for the repair duties of things. FreeBSD also adopted a more complex boot loader that would load the
> kernel, modules, set tunables, etc prior to kicking off the kernel. Between /rescue having all the tools needed
> to repair bad updates, repair failing disks, get that one last backup before the drive is dead while you wait
> for the new drive to be delivered, etc, and /boot/loader being able to script loading the kernel while the BIOS
> was still around so the need for drivers in the loader was lessened. However, as the BIOS evolved into UEFI
> and FreeBSD pushed into the embedded space whose firmware provided a less rich environment to the boot
> loader, so it was able to load things off fewer and fewer devices, it became clear that it would need a pivot
> root feature to allow it to boot all the way into FreeBSD, load some drivers from an included ram disk, and then
> use that mount a new root and then 'reroot' to that by killing everything and running init from that new root.
> FreeBSD also moved from Forth to Lua in its scripting language for the boot loader, giving 'pre boot'
> environment support better features. I also added the ability to use FreeBSD boot loader as a Linux binary
> to load FreeBSD and its metadata from a LinuxBoot environment. Finally, FreeBSD has 'spun out' and
> generalized the /rescue feature to allow creation of any 'BeastyBox' environment, similar to what you get
> in a busy box, or clone, environment. This environment, though, is meant in large part on both Linux and
> FreeBSD to be in constrained environments where a full install is prohibitive (even those that never pivot
> to something more, like ap, routers and nas boxes).
> 
> NetBSD retains many of the old BSD stand-alone programs that started on the vax. I've not studied things
> beyond noticing this. OpenBSD is similar. Their boot chain is a bit simpler than FreeBSD's, though there's
> noises about porting FreeBSD's boot there. There's a port of /boot/loader to illumos too, but I don't know
> if it is the default, or just available. So I'll not chat about it more.
> 
> So the original 'standalone' environment where you had one program running on a system has evolved
> into either a rich boot loader environment that lets one do a lot to decide what kernel to load, or towards
> having a minimal selection of unix programs faster and using /bin/sh or similar to do scripting. These
> reduced environments are often called standalone, though all they share just the name with the earlier
> 'stand' programs: they are full unix programs, but with reduced feature sets and 'linker magic' to package
> them in a way that's faster, smaller, etc (eg all in one binary). FreeBSD's boot loader is an outgrowth
> of the original standalone env, by way of a port of NetBSD's libsa. 
> 
> I suspect in the future, we'll see more and more of a trend for low-level init and then handing off to some
> built-in kernel (be it Linux, BSD-based (there's now kexec), or whatever) to reuse more of the vetted code
> rather than re-inventing Unix inside the boot loader (which is a valid criticism of FreeBSD's boot loader,
> though it's rich feature set is what you get for the complexity).
> 
> Does that answer the prompt? Should I try to make this into more of a retrospective paper and actually
> do the research on the areas I was hand-wavy about?
> 
> Warner

[TUHS] Re: Unix install & "standalone" package

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 4922 bytes --]

below..

On Tue, Sep 5, 2023 at 7:16 AM Paul Ruizendaal via TUHS <tuhs@tuhs.org>
wrote:

> One pitfall I would like to avoid in my own thinking is conflating
> “installing” and “booting”,
>
That is an excellent point.  I think that the real question comes back to
how 'cold' the 'system' is.

Booting is the process of making the processor 'live' including the loading
of the 'initial memory contents' (IBM used to call it IPL for
Initial Program Load), while 'installation' is setting up a 'system' which
'lives' in the peripherals as long-term storage.  When you 'Power-On' a
CPU, you often need to IPL a system even if the peripherals have been set
up [back in the day of core memory, you might not even have that as the OS
was not lost when power was removed].


> The starting point seems to be a setup where a small set of standalone
> programs is used to load or repair the bits, as was done for 16-bit unix.
>
You are solving the 'chicken and egg' issue.  Truth is Ken's famous
"Reflections on Trusting Trust"  comes to bear here.   The fact is you are
'setting up' (or restoring) a system for bits that were generated on
another. The critical point is that you are setting the initial 'systems'
bits into the peripherals for that specific system.   You are relying on
making copies of those bits from another system.  The question is how to
get them into the new media.

All the standalone system is doing is offering you a minimum way of making
a copy of that other system without having access to its hardware directly.



> A next conceptual step seems to be where first a very basic system is
> installed that is then used for further installation or for repair. This
> step seems to have come early, if this 32V install page is reflective of
> how it was done back in 1980:
> https://gunkies.org/wiki/Installing_32V_on_SIMH  The idea to use disk
> swap space for this also seems to have come early (and I suppose the
> concept lives on in “rescue partitions”).
>

Newer versions of the system setup scheme offer more and more features, as
the options of how you set up might be different from the origin system
become greater and greater. Modern OS implementations of the Unix
technologies now run on a much more comprehensive range of devices, and
frankly, 'IPL' much less might be set up from so many different types of
sources, it's not surprising that the IPL schemes and the system setup
scheme are a lot more sophisticated.
But remember, when you examine the past scheme, you must also consider the
constraints of the timeframe (particularly the economics of certain
choices).  It's not that you could not have built something like some of
today's schemes — it was expensive with respect to what was available and,
frankly, not wholly necessary.



>
> Another conceptual step might be where early installer phases run a
> different, smaller kernel (or even OS) than the one being installed. There
> seems to be much potential for a “wheel of reincarnation” here where as an
> installer grows large, a pre-installer is created to load the installer and
> then the pre-installer grows large, etc. For booting, this wheel seems to
> have turned about 5 times in current Linux. Installing that from scratch on
> a fully blank SBC (without prepping a removable disk on another computer)
> also appears to have 4 or 5 revolutions. That is one revolution every 5
> years for the 25 years from the late seventies to the early 2000’s.
>

The circle started long before that.  For instance, the boot/IPL got more
flexible (like Sam's autoconfiguration work developed for VAX).   Why?
Because the HW configurations had begun to branch and get bushy.  Even with
the PDP-11's peripheral set in V7, it was getting more complex.   But
those kernel
configurations were statically linked.  Sun added dynamic linking, making
both IPL, much less setup/installation more complex.   But frankly, to do
that on an earlier machine took a lot of work.   BSD2.9 did backport much
of the VAX work but not all of it.  It was often too difficult, and the
'gain' for the effort was low.

So my modern UNIX implementations, be it macOS, *BSD, or the Linux family,
the schemes have matured and been recreated over and over.  For instance,
you can look at what Apple's done with its system install scheme.   It
hides the setup/tear down as of the root and then makes a read-only root
system under the covers.   This is both a blessing and a curse.  Sure, it
helps make things a lot more secure for them and basic users, but that
choice breaks traditional admin scripts [which drives me nuts think -
/etc/periodic] because the new Apple system implementors don't understand
why the core system works the way it does and thus those of us that have
scripts have worked since V7 in the late 1970s, now don't.

Clem
ᐧ

[-- Attachment #2: Type: text/html, Size: 9204 bytes --]

[TUHS] Re: Unix install & "standalone" package

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 13663 bytes --]

On Tue, Sep 5, 2023 at 5:15 AM Paul Ruizendaal <pnr@planet.nl> wrote:

> > Does that answer the prompt? Should I try to make this into more of a
> retrospective paper and actually
> > do the research on the areas I was hand-wavy about?
>
> That certainly answered the prompt, much appreciated the walkthrough.
> Currently just trying to get a view of the field and to collect
> recollections on how it was done back in the day.
>
> Part of it is finding a conceptual framework that can make sense of it all.
>

Yes. V7 and other early unix had a 'stand alone' framework. So that's one
piece of the puzzle. It could be used for anything (I've seen private
diagnostic stand alone programs at Solbourne that would do more extensive
tests than the boot roms did for RMA and manufacturing line acceptance).
But as they come down to us from AT&T, they are just the bits needed to
install a system from scratch (or in some cases fix a system by booting the
distribution tapes).


> One pitfall I would like to avoid in my own thinking is conflating
> “installing” and “booting”, even though the two seem related (one loads
> bits into permanent storage, the other into volatile storage; both are
> built around incrementally adding capability). When variable hardware comes
> into play, the two mix even more. Also related is the topic of recovery.
>

There's "booting" and then there's "bootstrapping". booting happens all the
time, but installation happens rarely. The problem is that the install
process for V7 and earlier systems was to load a series of programs into
memory from tape that did each step of the process to stay within the 64k
limits imposed by the platform. That process was done by a series of
standalone programs that could, in some cases, act as system repair for
certain cases of 'afu' system.


> The starting point seems to be a setup where a small set of standalone
> programs is used to load or repair the bits, as was done for 16-bit unix.
>

Yea, that's where stand came from originally: An environment that could run
a limited program w/o booting the kernel...


> A next conceptual step seems to be where first a very basic system is
> installed that is then used for further installation or for repair. This
> step seems to have come early, if this 32V install page is reflective of
> how it was done back in 1980:
> https://gunkies.org/wiki/Installing_32V_on_SIMH  The idea to use disk
> swap space for this also seems to have come early (and I suppose the
> concept lives on in “rescue partitions”).
>

Yes. It was quite common for the next step to be when demand paging became
a thing. Once demand paging became a thing and you didn't need a dedicated
swap partition, you could 'borrow' the installed system's swap partition
(or what would become the installed system's swap partition) to have a
richer set of tools. This would use otherwise wasted space for the install,
then you'd overwrite it with whatever workload you put on your system.


> Another conceptual step might be where early installer phases run a
> different, smaller kernel (or even OS) than the one being installed. There
> seems to be much potential for a “wheel of reincarnation” here where as an
> installer grows large, a pre-installer is created to load the installer and
> then the pre-installer grows large, etc. For booting, this wheel seems to
> have turned about 5 times in current Linux. Installing that from scratch on
> a fully blank SBC (without prepping a removable disk on another computer)
> also appears to have 4 or 5 revolutions. That is one revolution every 5
> years for the 25 years from the late seventies to the early 2000’s.
>

Indeed, the whole linuxboot saga where you boot a minimal linux kernel that
has a shell script (or similar) that decides where to load the complete
kernel from is also interesting (doubly so to me because it means I can
sneak in at that point and load FreeBSD onto the system).


> Then there is the question of where in the "installer stack" to stop. My
> current interest excludes (precursors to) package managers, containers, etc.
>

Yes. For me, the 'preboot environment' is a better mental framework to look
at it with. Or 'pre-kernel' or 'non-kernel' environments. Though that
paradigm runs out of steam in the 90s when people started booting kernels
with bundled ram disks (in various flavors) to do the installation to allow
boot medium to be used to load additional data (eg the system). The
read/write nature of RAM made this vector useful, even when CDROMs appeared
on the scene. It was easier to run off a ram disk than off the CDROM
because you don't have to have symlinks to a (small) RAM disks for the bits
of the system that needed to be read/write during the install process...

The 'standalone' stuff, as typified by the src/stand directory, was all
pre-kernel or non-kernel use-cases. Sometimes it was used to build a bit of
code that had to be small, but was also used to build code to load the
kernel (even dating back to the original boot loader for V7, though I'm
relying on my memory which may be confusing that with the 2BSD
bootstrap)... The stuff that was 'standalone-but-on-a-unix-kernel' came
later.

It might also be amusing to note: I once contemplated porting V7 unix to
replace FreeBSD's stand environment to replace the various ad-hock hacks
that had grown up. But I found that those hacks were smaller in the end,
and needed less code space to support more things (excluding crypto and
compression) than the V7 did once you started looking at modifying it to
support multiple filesystem types, adding a network stack etc. While V7 is
nice and clean, it's also immature in terms of features needed for a modern
loader...

Warner


> In short: much to ponder, thanks to all for sharing recollections.
>
>
> > On 4 Sep 2023, at 19:07, Warner Losh <imp@bsdimp.com> wrote:
> >
> >
> >
> > On Mon, Sep 4, 2023 at 3:58 AM Paul Ruizendaal via TUHS <tuhs@tuhs.org>
> wrote:
> >
> > Recently, I was looking into the “Das U-Boot” boot loader package.
> Summarised with great simplification, u-boot bundles device drivers, file
> systems, commands and a Bourne-like shell into a standalone package.
> Normally it auto-runs a script that brings up a system, but when used in
> interactive mode it allows a great deal of poking around.
> >
> > It made me think of the “standalone” set of programs for installing
> early Unix. On 16-bit understandably each basic command has to be a
> separate standalone program, but after the shift to 32-bit bundling more
> functionality in a single binary would have become possible.
> >
> > How did the Unix “standalone” package evolve in the 80’s, both in the
> research and BSD lineages? Is there any retrospective paper about that? Or
> is it a case of “Use the source, Luke”?
> >
> > The stand package continued in research and BSD to be those programs
> needed to install and/or recover
> > badly damaged systems.  You could create a new file system, copy a file
> from the tape to a partition, etc.
> > You couldn't do general scripting with this, by and large.
> >
> > Originally, they were tape programs. This made sense because of its
> original focus. In time, some systems
> > could load the stand alone programs instead of the kernel, but they
> continued the original focus.
> >
> > This is, imho, due in large part due to the miniroot. The miniroot
> evolved into both a full-enough system
> > to do the installation scripts in shell instead of C (Venix, at least,
> had their install program written in C).
> > You'd copy the minroot to swap and then install the system. But a number
> of additional programs were
> > placed into the miniroot so you could do some limited filesystem repair,
> file editing, etc.
> >
> > In addition, many vendor's ROMs grew in complexity. Solbourne's ROMs,
> for example, could do basic
> > repair of UFS (clri level, not fsck level), and copy files from one
> place to another. I often recovered a
> > Solbourne system I screwed up by attaching an external SCSI drive that
> had a known good kernel,
> > init, etc.
> >
> > The 'stand' environment was a whole set of tools that could be used to
> build stand-alone programs that
> > shared much code of their full unix brethren, despite not having a full
> kernel under them. Kernel services
> > were provided by different libraries that did filesystem things, block
> driver things, network things, etc
> > in a similar way to Unix, but with a much reduced footprint.
> >
> > initramfs, as has been mentioned elsewhere, is pretty much a Linux
> invention. It was designed to
> > 'punt' on the choose where to load things from and have a very minimal
> interface between the boot
> > loader and the system. In time, it grew to support more interfaces, more
> ways of loading, and better
> > ways to mount something that you could then 'pivot' onto. Few other unix
> systems went this route, though
> > many adopted some variation on the pivot_root functionality. Linux has
> moved beyond the pivot root after
> > having booted the correct kernel into being able to take over the
> machine early in, say, UEFI startup with
> > a minimal kernel and initramfs that just knows how to load the next
> kernel. They skipped the complex boot
> > loader stage, and went straight to the 'run linux earlier' stage which
> is how things like LinuxBoot, coreboot
> > and others have put the boot logic into bash scripts. The ability to
> 'kexec' a kernel and replace the current
> > running kernel originated in the 'non-stop' world that wanted to reduce
> downtime. Now, it's used to reduce
> > firmware complexity by eliminating large swaths of UEFI from the boot
> process, but also generalizes in
> > the embedded space.
> >
> > FreeBSD, from around FreeBSD 2 (1995 or so), had /rescue which largely
> took over form the stand alone environment
> > for the repair duties of things. FreeBSD also adopted a more complex
> boot loader that would load the
> > kernel, modules, set tunables, etc prior to kicking off the kernel.
> Between /rescue having all the tools needed
> > to repair bad updates, repair failing disks, get that one last backup
> before the drive is dead while you wait
> > for the new drive to be delivered, etc, and /boot/loader being able to
> script loading the kernel while the BIOS
> > was still around so the need for drivers in the loader was lessened.
> However, as the BIOS evolved into UEFI
> > and FreeBSD pushed into the embedded space whose firmware provided a
> less rich environment to the boot
> > loader, so it was able to load things off fewer and fewer devices, it
> became clear that it would need a pivot
> > root feature to allow it to boot all the way into FreeBSD, load some
> drivers from an included ram disk, and then
> > use that mount a new root and then 'reroot' to that by killing
> everything and running init from that new root.
> > FreeBSD also moved from Forth to Lua in its scripting language for the
> boot loader, giving 'pre boot'
> > environment support better features. I also added the ability to use
> FreeBSD boot loader as a Linux binary
> > to load FreeBSD and its metadata from a LinuxBoot environment. Finally,
> FreeBSD has 'spun out' and
> > generalized the /rescue feature to allow creation of any 'BeastyBox'
> environment, similar to what you get
> > in a busy box, or clone, environment. This environment, though, is meant
> in large part on both Linux and
> > FreeBSD to be in constrained environments where a full install is
> prohibitive (even those that never pivot
> > to something more, like ap, routers and nas boxes).
> >
> > NetBSD retains many of the old BSD stand-alone programs that started on
> the vax. I've not studied things
> > beyond noticing this. OpenBSD is similar. Their boot chain is a bit
> simpler than FreeBSD's, though there's
> > noises about porting FreeBSD's boot there. There's a port of
> /boot/loader to illumos too, but I don't know
> > if it is the default, or just available. So I'll not chat about it more.
> >
> > So the original 'standalone' environment where you had one program
> running on a system has evolved
> > into either a rich boot loader environment that lets one do a lot to
> decide what kernel to load, or towards
> > having a minimal selection of unix programs faster and using /bin/sh or
> similar to do scripting. These
> > reduced environments are often called standalone, though all they share
> just the name with the earlier
> > 'stand' programs: they are full unix programs, but with reduced feature
> sets and 'linker magic' to package
> > them in a way that's faster, smaller, etc (eg all in one binary).
> FreeBSD's boot loader is an outgrowth
> > of the original standalone env, by way of a port of NetBSD's libsa.
> >
> > I suspect in the future, we'll see more and more of a trend for
> low-level init and then handing off to some
> > built-in kernel (be it Linux, BSD-based (there's now kexec), or
> whatever) to reuse more of the vetted code
> > rather than re-inventing Unix inside the boot loader (which is a valid
> criticism of FreeBSD's boot loader,
> > though it's rich feature set is what you get for the complexity).
> >
> > Does that answer the prompt? Should I try to make this into more of a
> retrospective paper and actually
> > do the research on the areas I was hand-wavy about?
> >
> > Warner
>
>

[-- Attachment #2: Type: text/html, Size: 15791 bytes --]