From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 31537 invoked from network); 5 Sep 2023 11:16:14 -0000 Received: from minnie.tuhs.org (50.116.15.146) by inbox.vuxu.org with ESMTPUTF8; 5 Sep 2023 11:16:14 -0000 Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id 27FF24102F; Tue, 5 Sep 2023 21:16:06 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuhs.org; s=dkim; t=1693912566; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-owner:list-unsubscribe:list-subscribe:list-post; bh=K+iYGjjsVAq+mX6eQvZP79D99heqIJ2y7ARrx8fuptE=; b=eOD0wKwiG21QJgA2xcDP39F2gNjsdXoJpF0iFd7oocrhFcbw36l6whTFUzO69asJ8p6+pg Z9u+KkwX88/UT1JxF1WpwuGAaRUnOM4386ySBO4oTou4eESTUMPk9FxGhtndu+UJCX/hsE N6QSsNrpgnZtozoRGvBKVFDIrWcvnZQ= Received: from ewsoutbound.kpnmail.nl (ewsoutbound.kpnmail.nl [195.121.94.169]) by minnie.tuhs.org (Postfix) with ESMTPS id 4778D41029 for ; Tue, 5 Sep 2023 21:15:45 +1000 (AEST) X-KPN-MessageId: 8904d771-4bdd-11ee-849b-005056abad63 Received: from smtp.kpnmail.nl (unknown [10.31.155.40]) by ewsoutbound.so.kpn.org (Halon) with ESMTPS id 8904d771-4bdd-11ee-849b-005056abad63; Tue, 05 Sep 2023 13:15:35 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=planet.nl; s=planet01; h=to:message-id:date:from:subject:mime-version:content-type; bh=K+iYGjjsVAq+mX6eQvZP79D99heqIJ2y7ARrx8fuptE=; b=keTklLuWa4QdgZrVZONjB2Z5I3u/9oITFKEaR1NGg1ysqlI1xMAz8/zUpDFDnocg8R4xU9YIhpWuc IhcMl6O09lcjJt+ARuJtsNNBX/xQJckxmORPxf1ZhOQxN8g8zd4s6Qq5UwlIb10FyqI1K7UTUYyfde n9OAzAXrC89VqEos= X-KPN-MID: 33|wo3yPYKkKc/UYJRz0KRxCgY0KHiH13HMkuLcXzZjrOMMinIbmDCg4VyQUtKnFuH +t7IN4/bMETg5z3ySmc2s1Z+BtenycKOaRGi1ZWULjo0= X-KPN-VerifiedSender: Yes X-CMASSUN: 33|nAs0wn+d8XvFhwHp91d/hHQEOskBf3Do748V8ySrTm0sSWJ/D/ILsY26b6MPGQC 5Qauooovty1AEghboZMr3rQ== X-Originating-IP: 77.172.38.96 Received: from smtpclient.apple (77-172-38-96.fixed.kpn.net [77.172.38.96]) by smtp.kpnmail.nl (Halon) with ESMTPSA id 88f602a4-4bdd-11ee-93b6-005056ab7584; Tue, 05 Sep 2023 13:15:35 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) In-Reply-To: Date: Tue, 5 Sep 2023 13:15:34 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: <8D99EBF7-CB1D-48EF-B6AD-914B45D48A5F@planet.nl> References: To: Warner Losh X-Mailer: Apple Mail (2.3654.120.0.1.13) Message-ID-Hash: V2BZOV3MSIX6PDCG6JEWRGCCKJY5XB5Z X-Message-ID-Hash: V2BZOV3MSIX6PDCG6JEWRGCCKJY5XB5Z X-MailFrom: pnr@planet.nl X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: "tuhs@tuhs.org" X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: Unix install & "standalone" package List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: Paul Ruizendaal via TUHS Reply-To: Paul Ruizendaal X-Spam: Yes > Does that answer the prompt? Should I try to make this into more of a = retrospective paper and actually > do the research on the areas I was hand-wavy about? That certainly answered the prompt, much appreciated the walkthrough. = Currently just trying to get a view of the field and to collect = recollections on how it was done back in the day. Part of it is finding a conceptual framework that can make sense of it = all. One pitfall I would like to avoid in my own thinking is conflating = =E2=80=9Cinstalling=E2=80=9D and =E2=80=9Cbooting=E2=80=9D, even though = the two seem related (one loads bits into permanent storage, the other = into volatile storage; both are built around incrementally adding = capability). When variable hardware comes into play, the two mix even = more. Also related is the topic of recovery. The starting point seems to be a setup where a small set of standalone = programs is used to load or repair the bits, as was done for 16-bit = unix. A next conceptual step seems to be where first a very basic system is = installed that is then used for further installation or for repair. This = step seems to have come early, if this 32V install page is reflective of = how it was done back in 1980: = https://gunkies.org/wiki/Installing_32V_on_SIMH The idea to use disk = swap space for this also seems to have come early (and I suppose the = concept lives on in =E2=80=9Crescue partitions=E2=80=9D). Another conceptual step might be where early installer phases run a = different, smaller kernel (or even OS) than the one being installed. = There seems to be much potential for a =E2=80=9Cwheel of = reincarnation=E2=80=9D here where as an installer grows large, a = pre-installer is created to load the installer and then the = pre-installer grows large, etc. For booting, this wheel seems to have = turned about 5 times in current Linux. Installing that from scratch on a = fully blank SBC (without prepping a removable disk on another computer) = also appears to have 4 or 5 revolutions. That is one revolution every 5 = years for the 25 years from the late seventies to the early 2000=E2=80=99s= . Then there is the question of where in the "installer stack" to stop. My = current interest excludes (precursors to) package managers, containers, = etc. In short: much to ponder, thanks to all for sharing recollections. > On 4 Sep 2023, at 19:07, Warner Losh wrote: >=20 >=20 >=20 > On Mon, Sep 4, 2023 at 3:58=E2=80=AFAM Paul Ruizendaal via TUHS = wrote: >=20 > Recently, I was looking into the =E2=80=9CDas U-Boot=E2=80=9D boot = loader package. Summarised with great simplification, u-boot bundles = device drivers, file systems, commands and a Bourne-like shell into a = standalone package. Normally it auto-runs a script that brings up a = system, but when used in interactive mode it allows a great deal of = poking around. >=20 > It made me think of the =E2=80=9Cstandalone=E2=80=9D set of programs = for installing early Unix. On 16-bit understandably each basic command = has to be a separate standalone program, but after the shift to 32-bit = bundling more functionality in a single binary would have become = possible. >=20 > How did the Unix =E2=80=9Cstandalone=E2=80=9D package evolve in the = 80=E2=80=99s, both in the research and BSD lineages? Is there any = retrospective paper about that? Or is it a case of =E2=80=9CUse the = source, Luke=E2=80=9D? >=20 > The stand package continued in research and BSD to be those programs = needed to install and/or recover > badly damaged systems. You could create a new file system, copy a = file from the tape to a partition, etc. > You couldn't do general scripting with this, by and large. >=20 > Originally, they were tape programs. This made sense because of its = original focus. In time, some systems > could load the stand alone programs instead of the kernel, but they = continued the original focus. >=20 > This is, imho, due in large part due to the miniroot. The miniroot = evolved into both a full-enough system > to do the installation scripts in shell instead of C (Venix, at least, = had their install program written in C). > You'd copy the minroot to swap and then install the system. But a = number of additional programs were > placed into the miniroot so you could do some limited filesystem = repair, file editing, etc. >=20 > In addition, many vendor's ROMs grew in complexity. Solbourne's ROMs, = for example, could do basic > repair of UFS (clri level, not fsck level), and copy files from one = place to another. I often recovered a > Solbourne system I screwed up by attaching an external SCSI drive that = had a known good kernel, > init, etc. >=20 > The 'stand' environment was a whole set of tools that could be used to = build stand-alone programs that > shared much code of their full unix brethren, despite not having a = full kernel under them. Kernel services > were provided by different libraries that did filesystem things, block = driver things, network things, etc > in a similar way to Unix, but with a much reduced footprint. >=20 > initramfs, as has been mentioned elsewhere, is pretty much a Linux = invention. It was designed to > 'punt' on the choose where to load things from and have a very minimal = interface between the boot > loader and the system. In time, it grew to support more interfaces, = more ways of loading, and better > ways to mount something that you could then 'pivot' onto. Few other = unix systems went this route, though > many adopted some variation on the pivot_root functionality. Linux has = moved beyond the pivot root after > having booted the correct kernel into being able to take over the = machine early in, say, UEFI startup with > a minimal kernel and initramfs that just knows how to load the next = kernel. They skipped the complex boot > loader stage, and went straight to the 'run linux earlier' stage which = is how things like LinuxBoot, coreboot > and others have put the boot logic into bash scripts. The ability to = 'kexec' a kernel and replace the current > running kernel originated in the 'non-stop' world that wanted to = reduce downtime. Now, it's used to reduce > firmware complexity by eliminating large swaths of UEFI from the boot = process, but also generalizes in > the embedded space. >=20 > FreeBSD, from around FreeBSD 2 (1995 or so), had /rescue which largely = took over form the stand alone environment > for the repair duties of things. FreeBSD also adopted a more complex = boot loader that would load the > kernel, modules, set tunables, etc prior to kicking off the kernel. = Between /rescue having all the tools needed > to repair bad updates, repair failing disks, get that one last backup = before the drive is dead while you wait > for the new drive to be delivered, etc, and /boot/loader being able to = script loading the kernel while the BIOS > was still around so the need for drivers in the loader was lessened. = However, as the BIOS evolved into UEFI > and FreeBSD pushed into the embedded space whose firmware provided a = less rich environment to the boot > loader, so it was able to load things off fewer and fewer devices, it = became clear that it would need a pivot > root feature to allow it to boot all the way into FreeBSD, load some = drivers from an included ram disk, and then > use that mount a new root and then 'reroot' to that by killing = everything and running init from that new root. > FreeBSD also moved from Forth to Lua in its scripting language for the = boot loader, giving 'pre boot' > environment support better features. I also added the ability to use = FreeBSD boot loader as a Linux binary > to load FreeBSD and its metadata from a LinuxBoot environment. = Finally, FreeBSD has 'spun out' and > generalized the /rescue feature to allow creation of any 'BeastyBox' = environment, similar to what you get > in a busy box, or clone, environment. This environment, though, is = meant in large part on both Linux and > FreeBSD to be in constrained environments where a full install is = prohibitive (even those that never pivot > to something more, like ap, routers and nas boxes). >=20 > NetBSD retains many of the old BSD stand-alone programs that started = on the vax. I've not studied things > beyond noticing this. OpenBSD is similar. Their boot chain is a bit = simpler than FreeBSD's, though there's > noises about porting FreeBSD's boot there. There's a port of = /boot/loader to illumos too, but I don't know > if it is the default, or just available. So I'll not chat about it = more. >=20 > So the original 'standalone' environment where you had one program = running on a system has evolved > into either a rich boot loader environment that lets one do a lot to = decide what kernel to load, or towards > having a minimal selection of unix programs faster and using /bin/sh = or similar to do scripting. These > reduced environments are often called standalone, though all they = share just the name with the earlier > 'stand' programs: they are full unix programs, but with reduced = feature sets and 'linker magic' to package > them in a way that's faster, smaller, etc (eg all in one binary). = FreeBSD's boot loader is an outgrowth > of the original standalone env, by way of a port of NetBSD's libsa.=20 >=20 > I suspect in the future, we'll see more and more of a trend for = low-level init and then handing off to some > built-in kernel (be it Linux, BSD-based (there's now kexec), or = whatever) to reuse more of the vetted code > rather than re-inventing Unix inside the boot loader (which is a valid = criticism of FreeBSD's boot loader, > though it's rich feature set is what you get for the complexity). >=20 > Does that answer the prompt? Should I try to make this into more of a = retrospective paper and actually > do the research on the areas I was hand-wavy about? >=20 > Warner