The Unix Heritage Society mailing list
 help / color / mirror / Atom feed
* [TUHS] Porting the SysIII kernel: boot, config & device drivers
@ 2022-12-30 18:25 Paul Ruizendaal
  2022-12-30 18:56 ` [TUHS] " Steve Nickolas
  2022-12-31 14:59 ` Dan Cross
  0 siblings, 2 replies; 26+ messages in thread
From: Paul Ruizendaal @ 2022-12-30 18:25 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

As mentioned in the first post on SysIII porting, I was surprised to see how much code was needed to initialise modern hardware and to load an OS. Of course, modern devices are much more capable than the ones of 40 years ago, so maybe my surprise is misplaced. It did raise an interest in the history of Unix system configuration though.

It would seem that 5th Edition already contained a configuration program that generated a few system tables and the ‘low.s’ file with interrupt vectors and alike. Although it steadily grew in sophistication, the approach appears still the same in SysIII. I suppose this is all in line with common practice of the era, with OS’s typically having a ’system generation kit' to combine the pre-linked OS kernel with device drivers and system tables.

SysIII also introduces the "var struct" and the “v” kernel variable that summarises some of the system configuration. I’m not sure whether it has roots in earlier Unix systems, it does not seem to originate from Research. I’m not sure what the point of this ‘v’ kernel variable was. Does anybody remember?

One could argue that one of the drivers of the success of CP/M in the 1970’s was due to its clear separation between the boot rom, BIOS and BDOS components. As far as I am aware,  Unix prior to 1985 did never attempt to separate the device drivers from the other kernel code. I am not very familiar with early Xenix, it could be that Microsoft had both the skill and the interest to separate Xenix in a standard binary (i.e. BDOS part) and a device driver binary (i.e. BIOS part). Maybe the differences in MMU for the machines of the early 80’s were such that a standard binary could not be done anyway and separating out the device drivers would serve no purpose. Once the PC became dominant, maybe the point became moot for MS.

It would seem that the next step for Unix in the area of boot, config and device drivers came with Sun’s OpenBoot in 1988 or so. This also appears to be the first appearance of device trees to describe the hardware to the bios and the kernel. Moreover, it would seem to me that OpenBoot is a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988 the IO hardware had become sufficiently complex and/or diverse to warrant a break from tradition?

Was there any other notable Unix work on better organising the boot process and the device drivers prior to OpenBoot? 


^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-30 18:25 [TUHS] Porting the SysIII kernel: boot, config & device drivers Paul Ruizendaal
@ 2022-12-30 18:56 ` Steve Nickolas
  2022-12-31 14:59 ` Dan Cross
  1 sibling, 0 replies; 26+ messages in thread
From: Steve Nickolas @ 2022-12-30 18:56 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1130 bytes --]

On Fri, 30 Dec 2022, Paul Ruizendaal wrote:

> One could argue that one of the drivers of the success of CP/M in the 
> 1970’s was due to its clear separation between the boot rom, BIOS and 
> BDOS components. As far as I am aware, Unix prior to 1985 did never 
> attempt to separate the device drivers from the other kernel code. I am 
> not very familiar with early Xenix, it could be that Microsoft had both 
> the skill and the interest to separate Xenix in a standard binary (i.e. 
> BDOS part) and a device driver binary (i.e. BIOS part). Maybe the 
> differences in MMU for the machines of the early 80’s were such that a 
> standard binary could not be done anyway and separating out the device 
> drivers would serve no purpose. Once the PC became dominant, maybe the 
> point became moot for MS.

Certainly Microsoft *did* have an operating system, as early as 1981, that 
had the concept of separated BIOS and BDOS, but they didn't write it, they 
bought it 🤪

That said, given that it existed in MS-DOS, I can't imagine it wouldn't 
have been impossible to also implement in Xenix...

-uso.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-30 18:25 [TUHS] Porting the SysIII kernel: boot, config & device drivers Paul Ruizendaal
  2022-12-30 18:56 ` [TUHS] " Steve Nickolas
@ 2022-12-31 14:59 ` Dan Cross
  2022-12-31 19:08   ` Clem Cole
  2022-12-31 20:02   ` Paul Ruizendaal
  1 sibling, 2 replies; 26+ messages in thread
From: Dan Cross @ 2022-12-31 14:59 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: The Eunuchs Hysterical Society

On Fri, Dec 30, 2022 at 1:26 PM Paul Ruizendaal <pnr@planet.nl> wrote:
>[snip]
> It would seem that the next step for Unix in the area of boot, config and device drivers came with Sun’s OpenBoot in 1988 or so. This also appears to be the first appearance of device trees to describe the hardware to the bios and the kernel. Moreover, it would seem to me that OpenBoot is a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988 the IO hardware had become sufficiently complex and/or diverse to warrant a break from tradition?
>
> Was there any other notable Unix work on better organising the boot process and the device drivers prior to OpenBoot?

I think that BSD supported autoconfiguration on the VAX well before
OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
revamped in 4.4.

SBI was/is an attempt to define a common interface to different
devices using RISC-V CPUs, but it's growing wildly and tending more
towards UEFI+ACPI than something like OpenBoot, which was much
simpler.

In general, the idea of a BIOS isn't terrible: provide an interface
that decouples the OS and hardware. But in practice, no one has come
up with a particularly good set of interfaces yet. Ironically,
BSD-style autoconfig might have been the best yet.

        - Dan C.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 14:59 ` Dan Cross
@ 2022-12-31 19:08   ` Clem Cole
  2022-12-31 21:10     ` Dan Cross
  2022-12-31 21:11     ` Paul Ruizendaal
  2022-12-31 20:02   ` Paul Ruizendaal
  1 sibling, 2 replies; 26+ messages in thread
From: Clem Cole @ 2022-12-31 19:08 UTC (permalink / raw)
  To: Dan Cross; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2152 bytes --]

On Sat, Dec 31, 2022 at 10:02 AM Dan Cross <crossd@gmail.com> wrote:

> I think that BSD supported autoconfiguration on the VAX well before
> OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
> revamped in 4.4.
>
I can verify that.   The PDP-10's had it first, then VMS.  During the VMS
vs UNIX war at ARPA, when Joy did the FASTVAX work, autoconfig
 first appeared.   Sam rewrote it with what eventually became 4.2 with his
configuration system.

At the point of the vendors like, DEC, DG, HP, then Masscomp, Apollo and
Sun all had their own boot systems.  For Vaxen that was ok, as the busses
were DEC defined, but folks like Xylogics, Emulex and such wanted to sell
boards for the Multibus, VME as well as Unibus or proprietary busses.   So
boot driver were something they had to offer in source, at least to their
OEM's [actually Masscomp (Paul Cantrell) wrote Xylogic's reference code for
a number of them and licenses them back, which Sun used later as it turns
out].

Larry and Rob G can comment on the OpenBoot stuff what was birthed later on
as a way to solve some of that problem.  IIRC, the SPARC may have been a
partial driver for that as offering boot rooms and driver for 68000 or 8086
based boards was not going to be helpful for that market.

>
> SBI was/is an attempt to define a common interface to different
> devices using RISC-V CPUs, but it's growing wildly and tending more
> towards UEFI+ACPI than something like OpenBoot, which was much
> simpler.
>
UEFI+ACPI got/gets a bad rep because of the original IBM BIOS
implementations.

>
> In general, the idea of a BIOS isn't terrible: provide an interface
> that decouples the OS and hardware.

Exactly - the idea is actually a good one.  But the problem was the BIOS
designed/implemented by HW people, but OS folks.  Things like concurrency
minimum use of the CPU was not in the higher order bits.



> But in practice, no one has come
> up with a particularly good set of interfaces yet. Ironically,
> BSD-style autoconfig might have been the best yet.
>
??Maybe because it was OS types who knew what the OS needed to
discover/report/deliver back from the HW.

[-- Attachment #2: Type: text/html, Size: 4177 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 14:59 ` Dan Cross
  2022-12-31 19:08   ` Clem Cole
@ 2022-12-31 20:02   ` Paul Ruizendaal
  2022-12-31 21:04     ` Warner Losh
                       ` (3 more replies)
  1 sibling, 4 replies; 26+ messages in thread
From: Paul Ruizendaal @ 2022-12-31 20:02 UTC (permalink / raw)
  To: Dan Cross, The Unix Historical Society



> On 31 Dec 2022, at 15:59, Dan Cross <crossd@gmail.com> wrote:
> 
> On Fri, Dec 30, 2022 at 1:26 PM Paul Ruizendaal <pnr@planet.nl> wrote:
>> [snip]
>> It would seem that the next step for Unix in the area of boot, config and device drivers came with Sun’s OpenBoot in 1988 or so. This also appears to be the first appearance of device trees to describe the hardware to the bios and the kernel. Moreover, it would seem to me that OpenBoot is a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988 the IO hardware had become sufficiently complex and/or diverse to warrant a break from tradition?
>> 
>> Was there any other notable Unix work on better organising the boot process and the device drivers prior to OpenBoot?
> 
> I think that BSD supported autoconfiguration on the VAX well before
> OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
> revamped in 4.4.

That is interesting. Are you referring to this:
https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/conf
https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/man/man8/config.8

Or is auto configuration something else?

> SBI was/is an attempt to define a common interface to different
> devices using RISC-V CPUs, but it's growing wildly and tending more
> towards UEFI+ACPI than something like OpenBoot, which was much
> simpler.
> 
> In general, the idea of a BIOS isn't terrible: provide an interface
> that decouples the OS and hardware. But in practice, no one has come
> up with a particularly good set of interfaces yet. Ironically,
> BSD-style autoconfig might have been the best yet.

When it comes to abstracting devices, it would seem to me that virtio has gotten quite some traction in the last decade.

Looking at both the systems of 40 years ago and the Risc-V SoC’s of last year, I could imagine something like:

- A simple SBI, like the Berkeley Boot Loader (BBL) without the proxy kernel & host interface
- Devices presented to the OS as virtio devices
- MMU abstraction somewhat similar in idea to that in SYSV/68 [1]

Together this might be a usable Unix BIOS that could have worked in the early 80’s. One could also think of it as a simple hypervisor for only one client. The remaining BBL functionality is not all that different from the content in mch.s on 16-bit Unix (e.g. floating point emulation for CPU’s that don’t have it in hardware). A virtio device is not all that different from the interface presented by e.g. PDP-11 ethernet devices (e.g. DELUA), the MMU abstraction was contemporary.

High end systems could have had device hardware that implemented the virtio interface directly, low end systems could use simpler hardware and use the BIOS to present this interface via software.


[1] From mmu.h in the SYSV/68 source tree:

/*
	@(#)mmu.h	2.26 
	This is the header file for the UNIX V/68 generic
	MMU interface. This provides the information that
	is used by the various routines that call:

	mmufork ()
	mmuexec ()
	mmuexit ()
	mmuread ()
	mmuwrite ()
	mmuattach ()
	mmudetach ()
	mmuregs ()
	mmualloc ()
	mmuinit ()
	mmuint ()

	The above routines and secondary routines called
	by them are contained in io/mmu1.c and io/mmu2.c.
*/





^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 20:02   ` Paul Ruizendaal
@ 2022-12-31 21:04     ` Warner Losh
  2022-12-31 21:41     ` Dan Cross
                       ` (2 subsequent siblings)
  3 siblings, 0 replies; 26+ messages in thread
From: Warner Losh @ 2022-12-31 21:04 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: The Unix Historical Society

[-- Attachment #1: Type: text/plain, Size: 4452 bytes --]

On Sat, Dec 31, 2022, 1:03 PM Paul Ruizendaal <pnr@planet.nl> wrote:

>
>
> > On 31 Dec 2022, at 15:59, Dan Cross <crossd@gmail.com> wrote:
> >
> > On Fri, Dec 30, 2022 at 1:26 PM Paul Ruizendaal <pnr@planet.nl> wrote:
> >> [snip]
> >> It would seem that the next step for Unix in the area of boot, config
> and device drivers came with Sun’s OpenBoot in 1988 or so. This also
> appears to be the first appearance of device trees to describe the hardware
> to the bios and the kernel. Moreover, it would seem to me that OpenBoot is
> a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988
> the IO hardware had become sufficiently complex and/or diverse to warrant a
> break from tradition?
> >>
> >> Was there any other notable Unix work on better organising the boot
> process and the device drivers prior to OpenBoot?
>

OpenBoot also had device drivers written in 4th for OS agnostic goodness.
It never caught on, though the concept lives on in AML in ACPI which aims
lower in its abstraction...

>
> > I think that BSD supported autoconfiguration on the VAX well before
> > OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
> > revamped in 4.4.
>
> That is interesting. Are you referring to this:
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/conf
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/man/man8/config.8
>
> Or is auto configuration something else?
>

Config == what is in the kernel with some strong hints of where to look for
root and some devices (depending on the busses involved).

Autoconf == discovering devices during early boot on the system (maybe with
hints from config) and connecting them to the right driver (later with a
bidding system for self identifying busses). This process allows GENERIC
kernels and for hardware changes.

The two are tightly coupled, but happen at different times. Config happens
once per build, autoconf happens once per boot (though latter day BSDs will
run some subset of this for module loads and device hotplug).

Warner

> SBI was/is an attempt to define a common interface to different
> > devices using RISC-V CPUs, but it's growing wildly and tending more
> > towards UEFI+ACPI than something like OpenBoot, which was much
> > simpler.
> >
> > In general, the idea of a BIOS isn't terrible: provide an interface
> > that decouples the OS and hardware. But in practice, no one has come
> > up with a particularly good set of interfaces yet. Ironically,
> > BSD-style autoconfig might have been the best yet.
>
> When it comes to abstracting devices, it would seem to me that virtio has
> gotten quite some traction in the last decade.
>
> Looking at both the systems of 40 years ago and the Risc-V SoC’s of last
> year, I could imagine something like:
>
> - A simple SBI, like the Berkeley Boot Loader (BBL) without the proxy
> kernel & host interface
> - Devices presented to the OS as virtio devices
> - MMU abstraction somewhat similar in idea to that in SYSV/68 [1]
>
> Together this might be a usable Unix BIOS that could have worked in the
> early 80’s. One could also think of it as a simple hypervisor for only one
> client. The remaining BBL functionality is not all that different from the
> content in mch.s on 16-bit Unix (e.g. floating point emulation for CPU’s
> that don’t have it in hardware). A virtio device is not all that different
> from the interface presented by e.g. PDP-11 ethernet devices (e.g. DELUA),
> the MMU abstraction was contemporary.
>
> High end systems could have had device hardware that implemented the
> virtio interface directly, low end systems could use simpler hardware and
> use the BIOS to present this interface via software.
>
>
> [1] From mmu.h in the SYSV/68 source tree:
>
> /*
>         @(#)mmu.h       2.26
>         This is the header file for the UNIX V/68 generic
>         MMU interface. This provides the information that
>         is used by the various routines that call:
>
>         mmufork ()
>         mmuexec ()
>         mmuexit ()
>         mmuread ()
>         mmuwrite ()
>         mmuattach ()
>         mmudetach ()
>         mmuregs ()
>         mmualloc ()
>         mmuinit ()
>         mmuint ()
>
>         The above routines and secondary routines called
>         by them are contained in io/mmu1.c and io/mmu2.c.
> */
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 5978 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 19:08   ` Clem Cole
@ 2022-12-31 21:10     ` Dan Cross
  2022-12-31 21:39       ` Clem Cole
  2022-12-31 22:38       ` Theodore Ts'o
  2022-12-31 21:11     ` Paul Ruizendaal
  1 sibling, 2 replies; 26+ messages in thread
From: Dan Cross @ 2022-12-31 21:10 UTC (permalink / raw)
  To: Clem Cole; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On Sat, Dec 31, 2022 at 2:08 PM Clem Cole <clemc@ccc.com> wrote:
> On Sat, Dec 31, 2022 at 10:02 AM Dan Cross <crossd@gmail.com> wrote:
>> [snip]
>> SBI was/is an attempt to define a common interface to different
>> devices using RISC-V CPUs, but it's growing wildly and tending more
>> towards UEFI+ACPI than something like OpenBoot, which was much
>> simpler.
>
> UEFI+ACPI got/gets a bad rep because of the original IBM BIOS implementations.

I'm going to push back on this slightly: UEFI+ACPI get a bad rap
because, well, they're really pretty bad. Oh sure, some things are
reasonable: the ACPI table formats aren't awful. But I've been inside
a couple of these now and phew golly, they stink pretty badly. The
code is poor quality, and encourages running blobs of really dubious
provenance. See below....

>> In general, the idea of a BIOS isn't terrible: provide an interface
>> that decouples the OS and hardware.
>
> Exactly - the idea is actually a good one.  But the problem was the BIOS designed/implemented by HW people, but OS folks.  Things like concurrency minimum use of the CPU was not in the higher order bits.
>
>> But in practice, no one has come
>> up with a particularly good set of interfaces yet. Ironically,
>> BSD-style autoconfig might have been the best yet.
>
> ??Maybe because it was OS types who knew what the OS needed to discover/report/deliver back from the HW.

Perhaps this is what it is, but I think taking a step back and looking
at the problem more generally, it's because they're mutated into
solving the wrong problem. Consider AML: isn't this something that
ought to be handled in, I don't know, a device driver? Yes, that
driver may need access to some platform-specific firmware to abstract
the details, but an entire virtual machine running random code to
provide abstractions for the firmware writers who are running,
basically, a parallel operating systems is a bit on the nose. This
goes way beyond OpenBoot's forth modules on option ROMs (which were
kind of a nifty idea for device discovery and such things).

Mothy Roscoe gave a really interesting keynote at OSDI'21:
https://www.youtube.com/watch?v=36myc8wQhLo

I love how he describes the interfaces we have now as having
"congealed." But in some ways, UEFI+ACPI are the antithesis of what
the OS should be doing.

        - Dan C.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 19:08   ` Clem Cole
  2022-12-31 21:10     ` Dan Cross
@ 2022-12-31 21:11     ` Paul Ruizendaal
  1 sibling, 0 replies; 26+ messages in thread
From: Paul Ruizendaal @ 2022-12-31 21:11 UTC (permalink / raw)
  To: Clem Cole; +Cc: The Eunuchs Hysterical Society


> On 31 Dec 2022, at 20:08, Clem Cole <clemc@ccc.com> wrote:
> 
> [snip]
> 
> In general, the idea of a BIOS isn't terrible: provide an interface
> that decouples the OS and hardware.
> Exactly - the idea is actually a good one.  But the problem was the BIOS designed/implemented by HW people, but OS folks.  Things like concurrency minimum use of the CPU was not in the higher order bits.

My limited experience of working with "board support packages" does support the idea that these are written by the hardware designers as a quick afterthought after completing the chip.



^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 21:10     ` Dan Cross
@ 2022-12-31 21:39       ` Clem Cole
  2022-12-31 21:52         ` Dan Cross
  2022-12-31 23:25         ` Dave Horsfall
  2022-12-31 22:38       ` Theodore Ts'o
  1 sibling, 2 replies; 26+ messages in thread
From: Clem Cole @ 2022-12-31 21:39 UTC (permalink / raw)
  To: Dan Cross; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 860 bytes --]

On Sat, Dec 31, 2022 at 4:11 PM Dan Cross <crossd@gmail.com> wrote:

>
> I'm going to push back on this slightly: UEFI+ACPI get a bad rap
> because, well, they're really pretty bad. Oh sure, some things are
> reasonable: the ACPI table formats aren't awful. But I've been inside
> a couple of these now and phew golly, they stink pretty badly.
>
Fair enough.   UEFI+ACPI started from BIOS and really should have been a
full re-think and write by senior OS people.

It was not.   The problem for us techies, is that doing that was not going
to save or make anyone $s.

The problem for management was they wanted to keep the old BIOS around
(that's what the customer wanted - cheapest path forward) and unless there
was a reason to break from tradition, they were not going too.   Apple
could (and did) but HP/Dell et al did not want too.
ᐧ

[-- Attachment #2: Type: text/html, Size: 1919 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 20:02   ` Paul Ruizendaal
  2022-12-31 21:04     ` Warner Losh
@ 2022-12-31 21:41     ` Dan Cross
  2023-01-01  3:08     ` Warner Losh
  2023-01-01  8:05     ` Jonathan Gray
  3 siblings, 0 replies; 26+ messages in thread
From: Dan Cross @ 2022-12-31 21:41 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: The Unix Historical Society

On Sat, Dec 31, 2022 at 3:02 PM Paul Ruizendaal <pnr@planet.nl> wrote:
> > On 31 Dec 2022, at 15:59, Dan Cross <crossd@gmail.com> wrote:
> > I think that BSD supported autoconfiguration on the VAX well before
> > OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
> > revamped in 4.4.
>
> That is interesting. Are you referring to this:
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/conf
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/man/man8/config.8
>
> Or is auto configuration something else?

Warner answered this better than I could. :-)

> > SBI was/is an attempt to define a common interface to different
> > devices using RISC-V CPUs, but it's growing wildly and tending more
> > towards UEFI+ACPI than something like OpenBoot, which was much
> > simpler.
> >
> > In general, the idea of a BIOS isn't terrible: provide an interface
> > that decouples the OS and hardware. But in practice, no one has come
> > up with a particularly good set of interfaces yet. Ironically,
> > BSD-style autoconfig might have been the best yet.
>
> When it comes to abstracting devices, it would seem to me that virtio has gotten quite some traction in the last decade.

Virtio is solving a very, very different problem: this is limited
paravirtualization of devices for virtual machines. The idea of virtio
devices is predicated on a hypervisor somewhere providing a matching
implementation that, in turn, plugs into whatever primitives the host
provides for device management. Put another way, devices don't expose
virtio interfaces directly (generally speaking), and you need
something somewhere to translate from the virtio interfaces a guest
sees to actual useful services that the guest makes use of via those
interfaces.

A host _could_ emulate something like an actual physical hardware
device (for some category of device) but it turns out that's often not
particularly efficient; for instance virtio-net tends to be somewhat
faster than an emulated e1000. Still, virtio has some problems; you
can program DMA to anywhere in the guest physical address space, which
can be expensive to handle without an exit from the guest context:
virtio-net isn't as fast as Google's gvnic.

> Looking at both the systems of 40 years ago and the Risc-V SoC’s of last year, I could imagine something like:
>
> - A simple SBI, like the Berkeley Boot Loader (BBL) without the proxy kernel & host interface

This, I think, is really the hard part. The issue is that systems are
configured a myriad of different ways depending on the platform,
vendor, particular combination of IO devices, etc. That makes figuring
out where the host OS image is and getting it loaded into memory a
real pain in the booty; not to mention on modern systems where you've
got to do memory training as part of initializing DRAM controllers and
the like. Not to mention power-sequencing, initializing IO buses and
all that business: who assigns PCI BARs? What if the kernel is on a
PCI device, like an NVMe SSD or similar? Turning on a modern CPU is
really a non-trivial exercise, let alone getting the OS loaded. Once
you get it into memory, though, _starting_ the kernel is comparatively
easy. But all the gritty details before that are how we ended up with
enormous systems like UEFI.

> - Devices presented to the OS as virtio devices

But backed by what?

> - MMU abstraction somewhat similar in idea to that in SYSV/68 [1]

It's not clear to me what this would buy you; the MMU tends to be
architecturally defined, in a way that e.g. the boot process, IO bus
initialization, and even CPU topology discovery are not. And then
there are soft-TLBs (which are a bad idea, but that's a separate
discussion). I mean, most VM systems since SunOS 4 have a HAL and a
portable part already.

> Together this might be a usable Unix BIOS that could have worked in the early 80’s. One could also think of it as a simple hypervisor for only one client.

I believe that AIX on the PC/RT did basically this. Certainly, this
was true on the 370. The PALcode stuff on the Alpha was similar in
terms of having a BIOS-like abstraction with specific implementations
for Windows or VMS/Unix.

But there's an argument that this is functionality that _belongs_ in
the OS, which was the traditional Unix approach. Could it have been
done differently? I guess so, but would that have been a good system
design? I don't know.

> The remaining BBL functionality is not all that different from the content in mch.s on 16-bit Unix (e.g. floating point emulation for CPU’s that don’t have it in hardware). A virtio device is not all that different from the interface presented by e.g. PDP-11 ethernet devices (e.g. DELUA), the MMU abstraction was contemporary.
>
> High end systems could have had device hardware that implemented the virtio interface directly,

This is challenging with virtio because some devices basically rely on
the guest trapping into the host with a particular exit code to kick
off IO. I suppose you could do that with a shared semaphore, but then
you're burning cycles somewhere polling. You could interrupt yourself,
either synchronously or with a self-IPI, but then the interrupt
handler has to run the virtio handling code. In any event, it's no
longer a simple matter of writing to a device register somewhere to
make the "device" actually do something. Sadly, that's pretty baked
into the interface.

> low end systems could use simpler hardware and use the BIOS to present this interface via software.

But you've got to have some CPU run that software somewhere; either
you've got a satellite processor doing it or you're stealing cycles
from the OS. The latter is how a BIOS traditionally works and I
suspect that if we were doing this in the 1980s, having a dedicated IO
processor would have added non-trivial cost to a system, for little
gain: let's remember that Unix was _very_ success on a variety of
systems in that era without relying on a BIOS-type abstraction!

> [1] From mmu.h in the SYSV/68 source tree:
>
> /*
>         @(#)mmu.h       2.26
>         This is the header file for the UNIX V/68 generic
>         MMU interface. This provides the information that
>         is used by the various routines that call:
>
>         mmufork ()
>         mmuexec ()
>         mmuexit ()
>         mmuread ()
>         mmuwrite ()
>         mmuattach ()
>         mmudetach ()
>         mmuregs ()
>         mmualloc ()
>         mmuinit ()
>         mmuint ()
>
>         The above routines and secondary routines called
>         by them are contained in io/mmu1.c and io/mmu2.c.
> */
>
>
>
>

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 21:39       ` Clem Cole
@ 2022-12-31 21:52         ` Dan Cross
  2022-12-31 23:25         ` Dave Horsfall
  1 sibling, 0 replies; 26+ messages in thread
From: Dan Cross @ 2022-12-31 21:52 UTC (permalink / raw)
  To: Clem Cole; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On Sat, Dec 31, 2022 at 4:39 PM Clem Cole <clemc@ccc.com> wrote:
> On Sat, Dec 31, 2022 at 4:11 PM Dan Cross <crossd@gmail.com> wrote:
>> I'm going to push back on this slightly: UEFI+ACPI get a bad rap
>> because, well, they're really pretty bad. Oh sure, some things are
>> reasonable: the ACPI table formats aren't awful. But I've been inside
>> a couple of these now and phew golly, they stink pretty badly.
>
> Fair enough.   UEFI+ACPI started from BIOS and really should have been a full re-think and write by senior OS people.

Agreed.

> It was not.   The problem for us techies, is that doing that was not going to save or make anyone $s.
>
> The problem for management was they wanted to keep the old BIOS around (that's what the customer wanted - cheapest path forward) and unless there was a reason to break from tradition, they were not going too.   Apple could (and did) but HP/Dell et al did not want too.

Yeah. We canc'd it from our product (Oxide machines boot directly from
the x86 reset vector and go almost immediately into Unix; no
BIOS/UEFI+ACPI at all! Of course, there's a bunch of stuff that
happens before the x86 cores come out of reset, but that's a different
kettle of fish). We can get away with this because we control the
hardware and the host software, and the unit of compute our customers
interact with is a virtual machine, not the bare metal. When you
control the horizontal _and_ the vertical, you can put it all in the
host OS, where it arguably belongs. But in a world where software and
hardware are completely decoupled? I don't know how to make the
holistic boot approach work without a lot of system-specific code.
It's obvious that you need some kind of system interface, and for most
vendors, UEFI+ACPI is there and supported by the chip and firmware
vendors, so it's the default. No one ever got fired for using
UEFI/ACPI, I guess.

        - Dan C.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 21:10     ` Dan Cross
  2022-12-31 21:39       ` Clem Cole
@ 2022-12-31 22:38       ` Theodore Ts'o
  2022-12-31 22:55         ` Marc Donner
                           ` (2 more replies)
  1 sibling, 3 replies; 26+ messages in thread
From: Theodore Ts'o @ 2022-12-31 22:38 UTC (permalink / raw)
  To: Dan Cross; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On Sat, Dec 31, 2022 at 04:10:47PM -0500, Dan Cross wrote:
> >> But in practice, no one has come
> >> up with a particularly good set of interfaces yet. Ironically,
> >> BSD-style autoconfig might have been the best yet.
> >
> > ??Maybe because it was OS types who knew what the OS needed to discover/report/deliver back from the HW.
> 
> Perhaps this is what it is, but I think taking a step back and looking
> at the problem more generally, it's because they're mutated into
> solving the wrong problem.

More generally, we need to make sure we're all on the same page with
respect to requirements.  What are we trying to do with respect to
abstraction?  Do we just want to inform the OS about Port and MMIO
addresses, interrupts, etc, with the device driver living in the OS?
Or are are we trying to move the entire device driver plus bus
configuration/attachment probing into some kind of separate firmware
layer which is decoupled from the rest of the OS?

How important is portability?  Does this hardware abstraction layer
and/or firmware need to work with major legacy OS's such as Windows 95
(still in use by the US Government)?  Windows 10?  NetBSD?  Linux?
Or just only new research OS's?

How important is performance?  And does it need to support OS's that
might want to support interesting features, such as say, confidential
computing?  IOMMU's?  DMA from a storage device directly into memory
which is accessable over RDMA via Infiniband or iWARP or to GPU
memory?

Or are we just trying to solve the problem of loading the OS, or maybe
just initial bootstrap portion of the OS, where performance or support
for more exotic memory use cases might not be as important?

If we don't agree on exactly what problem that we are trying to solve
with BSD-style autoconfig, vs UEFI vs ACPI vs the x86 BIOS vs CP/M's
BIOS, etc.  Otherwise, the risk that we'll end up talking past one
another, or delivering a solution that we're happy with, but others
are not, is quite high IMHO.

						- Ted

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 22:38       ` Theodore Ts'o
@ 2022-12-31 22:55         ` Marc Donner
  2023-01-01  3:55         ` Dan Cross
  2023-01-01 20:29         ` Paul Ruizendaal
  2 siblings, 0 replies; 26+ messages in thread
From: Marc Donner @ 2022-12-31 22:55 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2546 bytes --]

This is not precisely what I remember, but I remember at USENIX when there
was a talk about the Open Boot work, someone sang a song.  I looked for the
song, but only found this page which doesn't read right to me:
https://everything2.com/title/The+Open+Firmware+Song
=====
nygeek.net
mindthegapdialogs.com/home <https://www.mindthegapdialogs.com/home>


On Sat, Dec 31, 2022 at 5:38 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Sat, Dec 31, 2022 at 04:10:47PM -0500, Dan Cross wrote:
> > >> But in practice, no one has come
> > >> up with a particularly good set of interfaces yet. Ironically,
> > >> BSD-style autoconfig might have been the best yet.
> > >
> > > ??Maybe because it was OS types who knew what the OS needed to
> discover/report/deliver back from the HW.
> >
> > Perhaps this is what it is, but I think taking a step back and looking
> > at the problem more generally, it's because they're mutated into
> > solving the wrong problem.
>
> More generally, we need to make sure we're all on the same page with
> respect to requirements.  What are we trying to do with respect to
> abstraction?  Do we just want to inform the OS about Port and MMIO
> addresses, interrupts, etc, with the device driver living in the OS?
> Or are are we trying to move the entire device driver plus bus
> configuration/attachment probing into some kind of separate firmware
> layer which is decoupled from the rest of the OS?
>
> How important is portability?  Does this hardware abstraction layer
> and/or firmware need to work with major legacy OS's such as Windows 95
> (still in use by the US Government)?  Windows 10?  NetBSD?  Linux?
> Or just only new research OS's?
>
> How important is performance?  And does it need to support OS's that
> might want to support interesting features, such as say, confidential
> computing?  IOMMU's?  DMA from a storage device directly into memory
> which is accessable over RDMA via Infiniband or iWARP or to GPU
> memory?
>
> Or are we just trying to solve the problem of loading the OS, or maybe
> just initial bootstrap portion of the OS, where performance or support
> for more exotic memory use cases might not be as important?
>
> If we don't agree on exactly what problem that we are trying to solve
> with BSD-style autoconfig, vs UEFI vs ACPI vs the x86 BIOS vs CP/M's
> BIOS, etc.  Otherwise, the risk that we'll end up talking past one
> another, or delivering a solution that we're happy with, but others
> are not, is quite high IMHO.
>
>                                                 - Ted
>

[-- Attachment #2: Type: text/html, Size: 3455 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 21:39       ` Clem Cole
  2022-12-31 21:52         ` Dan Cross
@ 2022-12-31 23:25         ` Dave Horsfall
  2023-01-01  1:02           ` Rob Pike
  1 sibling, 1 reply; 26+ messages in thread
From: Dave Horsfall @ 2022-12-31 23:25 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

On Sat, 31 Dec 2022, Clem Cole wrote:

[...]

> It was not.   The problem for us techies, is that doing that was not 
> going to save or make anyone $s.

Unfortunately, companies are run by marketoids; now, if the engineers were 
in charge...

[ I know; they'd go broke :-(  Marketoids have their uses, after all; many 
  a great Aussie company failed (Applix, Fairlight, etc) because the 
  product was not marketed properly. ]

-- Dave

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 23:25         ` Dave Horsfall
@ 2023-01-01  1:02           ` Rob Pike
  2023-01-01  1:16             ` George Michaelson
  2023-01-01  1:24             ` Larry McVoy
  0 siblings, 2 replies; 26+ messages in thread
From: Rob Pike @ 2023-01-01  1:02 UTC (permalink / raw)
  To: Dave Horsfall; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2939 bytes --]

A theory about organizations, inspired by OpenBoot.

As a company grows, which capitalism says it must, jobs change from
something someone needs to do from time to time, to something a person is
hired to do, to something a whole organization does. There are countless
examples, but OpenBoot serves as an illustration. Like some others on this
list, I'm sure, I've written a number of boot ROMs over the years, in the
service of a project for which a bootable system was a prerequisite. Not
hard, a day or two's work followed by some maintenance. So when I first saw
a major software project in the form of a boot ROM, which may well have
been my first taste of OpenBoot (it was a Sun thing, I'm pretty sure) I was
bemused.

But over time I've come to understand. As Sun grew (if I have the wrong
company, I apologize, but it's endemic in any industry, and if I have the
details of OpenBoot wrong, ditto), for various reasons the maintenance of
the boot ROM became more time consuming, until eventually it became
someone's full job, and then the boot ROM department's job. But allocating
people's time to tasks is an inexact process, plus once one's full focus
becomes boot ROMs, one's head fills with ideas. If one works on boot ROMs,
every problem one thinks about becomes one the boot ROM can solve or at
least help. (This particular disease is a major infection for compiler
teams.) And so the boot ROM grows and grows and grows, accumulating
features that are fun for the team to work on, but in the high-altitude
view not really worth it. As I said, though, it's really about allocating
people's time. "I have nothing to do at the moment, why don't I put a FORTH
interpreter into the boot ROM? And then I'll make it an industry standard.
Anyway it's my job to do boot ROMs so what else should I do now?" But you
wouldn't have done that if the boot ROM stayed at the level of the stack it
should have, doing the minimum necessary to get the operating system up and
running to let it do the heavy lifting. I never needed a FORTH interpreter
in my boot ROM. Maybe some thought they did but did they really? And was it
the right use of resources to put it there?

This is what happens in organizations. Employee performance evaluation at
Google followed this same arc until the need to keep the organization
responsible busy generated a process that placed a staggeringly expensive
tax on the rest of the company, to the point that a secondary tax was paid
in trying to figure out how to deal with the primary tax.

This then is my theory of capitalism: Things grow until they break into
pieces that must each be fed individually, triggering more growth that
eventually become tumors that sap the strength of the organization. Small
companies can do well because they haven't grown big enough yet to face
this problem. Not an original observation, but perhaps no one has used
OpenBoot as its exemplar before.

Happy New Year.

-rob


But

[-- Attachment #2: Type: text/html, Size: 4203 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01  1:02           ` Rob Pike
@ 2023-01-01  1:16             ` George Michaelson
  2023-01-01  1:40               ` Larry McVoy
  2023-01-01  1:24             ` Larry McVoy
  1 sibling, 1 reply; 26+ messages in thread
From: George Michaelson @ 2023-01-01  1:16 UTC (permalink / raw)
  To: Rob Pike; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1307 bytes --]

A counter argument which will be well understood as self-justifying if made
by a boot rom specialist:

Every machine I make winds up looking a bit different. The new bus has
different logic. The chip initialisation differs. Blobs become more
interestingly hard to handle because associated pre boot initialisation
dependency keeps rising and no amount of push back from me stops it.

If I make my boot ROM forth, I can reduce my marginal costs to writing
forth code for most variant handling and occasional uplift of new
primitives and constants into the forth for edge cases. My life gets
simpler if I implement the wheel of life.

I would imagine after the 10th sub variant, one would wind up thinking like
this.

Of course a rational alternative is to maintain a monrepo of all the
variants and recompile all of them all the time to make all the boot ROMs
far smaller. But making the generic anything ROM and changing only some
forth would be attractive.

 Never owned this problem. I did work with two groups doing lsi-11 images
for x.25 handling on yorkbox, and they definitely thought more like you
than me on this: hand code it, code it well, they aren't general purpose
devices when doing this kind of job. (I annoyed them a lot which tends to
"probably they were right" in hindsight on my part)

G

[-- Attachment #2: Type: text/html, Size: 1621 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01  1:02           ` Rob Pike
  2023-01-01  1:16             ` George Michaelson
@ 2023-01-01  1:24             ` Larry McVoy
  1 sibling, 0 replies; 26+ messages in thread
From: Larry McVoy @ 2023-01-01  1:24 UTC (permalink / raw)
  To: Rob Pike; +Cc: The Eunuchs Hysterical Society

On Sun, Jan 01, 2023 at 12:02:12PM +1100, Rob Pike wrote:
> A theory about organizations, inspired by OpenBoot.
> 
> "I have nothing to do at the moment, why don't I put a FORTH
> interpreter into the boot ROM? And then I'll make it an industry standard.

Yeah, that was Mitch's baby and you have the details as right as I can
remember.

I barfed at Forth, I've coded in Forth and didn't care for it.  I didn't
like lisp either but I could sort of see why some people get all worked
up for lisp, wasn't for me, but I could see the point.  I never saw the
point of Forth for anything.

But, for whatever reason, Mitch did.  Yuck.  The one time it was useful
in my memory, was during bringup and the kernel debugger wasn't working
yet.  So Mitch, or someone, coded up forth code to walk kernel data
structures, a set of a dump_$whatever.  Briefly useful.

Over all, I agree with everything Rob said.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01  1:16             ` George Michaelson
@ 2023-01-01  1:40               ` Larry McVoy
  2023-01-01  2:29                 ` Warner Losh
  0 siblings, 1 reply; 26+ messages in thread
From: Larry McVoy @ 2023-01-01  1:40 UTC (permalink / raw)
  To: George Michaelson; +Cc: The Eunuchs Hysterical Society

All true except for the Forth choice.  It's as bad, maybe worse, as 
choosing Tcl for your language.  I've written a ton of Tcl but I
need the Tk GUI part so I put up with Tcl to get it.  I'd never 
push Tcl as a language that other people had to use.  Same thing
with Forth.

I dunno what I'd pick, Perl in the old days, Python now (not that
I care for Python but everyone can program it).  Just pick something
that is trivial for someone to pick up.

On Sun, Jan 01, 2023 at 11:16:16AM +1000, George Michaelson wrote:
> A counter argument which will be well understood as self-justifying if made
> by a boot rom specialist:
> 
> Every machine I make winds up looking a bit different. The new bus has
> different logic. The chip initialisation differs. Blobs become more
> interestingly hard to handle because associated pre boot initialisation
> dependency keeps rising and no amount of push back from me stops it.
> 
> If I make my boot ROM forth, I can reduce my marginal costs to writing
> forth code for most variant handling and occasional uplift of new
> primitives and constants into the forth for edge cases. My life gets
> simpler if I implement the wheel of life.
> 
> I would imagine after the 10th sub variant, one would wind up thinking like
> this.
> 
> Of course a rational alternative is to maintain a monrepo of all the
> variants and recompile all of them all the time to make all the boot ROMs
> far smaller. But making the generic anything ROM and changing only some
> forth would be attractive.
> 
>  Never owned this problem. I did work with two groups doing lsi-11 images
> for x.25 handling on yorkbox, and they definitely thought more like you
> than me on this: hand code it, code it well, they aren't general purpose
> devices when doing this kind of job. (I annoyed them a lot which tends to
> "probably they were right" in hindsight on my part)
> 
> G

-- 
---
Larry McVoy           Retired to fishing          http://www.mcvoy.com/lm/boat

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01  1:40               ` Larry McVoy
@ 2023-01-01  2:29                 ` Warner Losh
  0 siblings, 0 replies; 26+ messages in thread
From: Warner Losh @ 2023-01-01  2:29 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2365 bytes --]

FreeBSD's boot loader uses Forth as it's scripting language.... well, we
replaced it a couple years ago with Lua... it's just big enough to be
useful and small enough to fit in BIOS constrained (640k) environments.
Perl and python are too big...

Warner


On Sat, Dec 31, 2022, 6:40 PM Larry McVoy <lm@mcvoy.com> wrote:

> All true except for the Forth choice.  It's as bad, maybe worse, as
> choosing Tcl for your language.  I've written a ton of Tcl but I
> need the Tk GUI part so I put up with Tcl to get it.  I'd never
> push Tcl as a language that other people had to use.  Same thing
> with Forth.
>
> I dunno what I'd pick, Perl in the old days, Python now (not that
> I care for Python but everyone can program it).  Just pick something
> that is trivial for someone to pick up.
>
> On Sun, Jan 01, 2023 at 11:16:16AM +1000, George Michaelson wrote:
> > A counter argument which will be well understood as self-justifying if
> made
> > by a boot rom specialist:
> >
> > Every machine I make winds up looking a bit different. The new bus has
> > different logic. The chip initialisation differs. Blobs become more
> > interestingly hard to handle because associated pre boot initialisation
> > dependency keeps rising and no amount of push back from me stops it.
> >
> > If I make my boot ROM forth, I can reduce my marginal costs to writing
> > forth code for most variant handling and occasional uplift of new
> > primitives and constants into the forth for edge cases. My life gets
> > simpler if I implement the wheel of life.
> >
> > I would imagine after the 10th sub variant, one would wind up thinking
> like
> > this.
> >
> > Of course a rational alternative is to maintain a monrepo of all the
> > variants and recompile all of them all the time to make all the boot ROMs
> > far smaller. But making the generic anything ROM and changing only some
> > forth would be attractive.
> >
> >  Never owned this problem. I did work with two groups doing lsi-11 images
> > for x.25 handling on yorkbox, and they definitely thought more like you
> > than me on this: hand code it, code it well, they aren't general purpose
> > devices when doing this kind of job. (I annoyed them a lot which tends to
> > "probably they were right" in hindsight on my part)
> >
> > G
>
> --
> ---
> Larry McVoy           Retired to fishing
> http://www.mcvoy.com/lm/boat
>

[-- Attachment #2: Type: text/html, Size: 3051 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 20:02   ` Paul Ruizendaal
  2022-12-31 21:04     ` Warner Losh
  2022-12-31 21:41     ` Dan Cross
@ 2023-01-01  3:08     ` Warner Losh
  2023-01-01  4:40       ` Dan Cross
  2023-01-01  8:05     ` Jonathan Gray
  3 siblings, 1 reply; 26+ messages in thread
From: Warner Losh @ 2023-01-01  3:08 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: The Unix Historical Society

[-- Attachment #1: Type: text/plain, Size: 6199 bytes --]

On Sat, Dec 31, 2022 at 1:03 PM Paul Ruizendaal <pnr@planet.nl> wrote:

>
>
> > On 31 Dec 2022, at 15:59, Dan Cross <crossd@gmail.com> wrote:
> >
> > On Fri, Dec 30, 2022 at 1:26 PM Paul Ruizendaal <pnr@planet.nl> wrote:
> >> [snip]
> >> It would seem that the next step for Unix in the area of boot, config
> and device drivers came with Sun’s OpenBoot in 1988 or so. This also
> appears to be the first appearance of device trees to describe the hardware
> to the bios and the kernel. Moreover, it would seem to me that OpenBoot is
> a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988
> the IO hardware had become sufficiently complex and/or diverse to warrant a
> break from tradition?
> >>
> >> Was there any other notable Unix work on better organising the boot
> process and the device drivers prior to OpenBoot?
> >
> > I think that BSD supported autoconfiguration on the VAX well before
> > OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
> > revamped in 4.4.
>
> That is interesting. Are you referring to this:
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/conf
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/man/man8/config.8
>
> Or is auto configuration something else?
>
> > SBI was/is an attempt to define a common interface to different
> > devices using RISC-V CPUs, but it's growing wildly and tending more
> > towards UEFI+ACPI than something like OpenBoot, which was much
> > simpler.
> >
> > In general, the idea of a BIOS isn't terrible: provide an interface
> > that decouples the OS and hardware. But in practice, no one has come
> > up with a particularly good set of interfaces yet. Ironically,
> > BSD-style autoconfig might have been the best yet.
>
> When it comes to abstracting devices, it would seem to me that virtio has
> gotten quite some traction in the last decade.
>
> Looking at both the systems of 40 years ago and the Risc-V SoC’s of last
> year, I could imagine something like:
>
> - A simple SBI, like the Berkeley Boot Loader (BBL) without the proxy
> kernel & host interface
> - Devices presented to the OS as virtio devices
> - MMU abstraction somewhat similar in idea to that in SYSV/68 [1]
>

Let's not forget that the diversity of systems is 1000s of times greater
than it was in the early days of Unix. There were just two busses (Unibus
and QBUS) in pdp-11 unix. There were a limited number of peripherals that
could live at a limited number of CSRs and compatibility requirements of
other DEC OSes meant that most of the 3rd party cards were compatible with
DEC gear, though there were exceptions. There was no hot plug, there was no
need to hierarchically assign address pools for things like PCIe hot-plug.
There was no USB. There were only a very limited number of keyboard and
graphics configurations. VAXen increased the number, but not by a huge
amount, at least in the early days before new busses were introduced).

As such, the unixes from the system III era simply didn't have any notion
of this diversity. So trying to layer it in will be quite difficult. The
SoC space has become huge, with memory being mapped in different locations,
busses being different, etc.

There's been much nasty said about FDT and ACPI, but they do solve real
problems: how to enumerate this diversity to the OS in a way that's sane
and might not always be as simple as returning a specific number but that
requires hardware access to answer even basic questions (because, say, the
CPUs were wired this way or that and you have to read those wirings).
Linux, even in Linuxboot environments, still uses ACPI, FDT and UEFI to get
the job done, and the code there isn't horrific.

V7 unix for the PDP-11 shipped with maybe 25 drivers total for the whole
system, and many of them were quite niche...


> Together this might be a usable Unix BIOS that could have worked in the
> early 80’s. One could also think of it as a simple hypervisor for only one
> client. The remaining BBL functionality is not all that different from the
> content in mch.s on 16-bit Unix (e.g. floating point emulation for CPU’s
> that don’t have it in hardware). A virtio device is not all that different
> from the interface presented by e.g. PDP-11 ethernet devices (e.g. DELUA),
> the MMU abstraction was contemporary.
>

virtio solves a different problem, though: It's goal is to provide THE
interface for mass storage, THE interface for networking, etc so that
hypervisor clients can limit their drivers substantially and not have to
deal with the thousands of drivers normally needed. ACPI/FDT just try to
make the non-self-describing aspects of the hardware described. Which is a
different target market (so that one can hook up the right driver of the
thousands to choose from when the OS autoconfig's the devices by whatever
means it does that, even Linux and System V have this concept, though with
radically different code than the BSDs).


> High end systems could have had device hardware that implemented the
> virtio interface directly, low end systems could use simpler hardware and
> use the BIOS to present this interface via software.
>

I'd worry that 'use the BIOS' bit here leads to the ugly UEFI/ACPI hybrid
to be generic enough to describe everything.

Now, I don't disagree with the org chart args for why they are so large
outside of linuxboot, but they do fill a vacuum that would otherwise exist.

Warner


>
> [1] From mmu.h in the SYSV/68 source tree:
>
> /*
>         @(#)mmu.h       2.26
>         This is the header file for the UNIX V/68 generic
>         MMU interface. This provides the information that
>         is used by the various routines that call:
>
>         mmufork ()
>         mmuexec ()
>         mmuexit ()
>         mmuread ()
>         mmuwrite ()
>         mmuattach ()
>         mmudetach ()
>         mmuregs ()
>         mmualloc ()
>         mmuinit ()
>         mmuint ()
>
>         The above routines and secondary routines called
>         by them are contained in io/mmu1.c and io/mmu2.c.
> */
>
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 7712 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 22:38       ` Theodore Ts'o
  2022-12-31 22:55         ` Marc Donner
@ 2023-01-01  3:55         ` Dan Cross
  2023-01-01 20:29         ` Paul Ruizendaal
  2 siblings, 0 replies; 26+ messages in thread
From: Dan Cross @ 2023-01-01  3:55 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: Paul Ruizendaal, The Eunuchs Hysterical Society

On Sat, Dec 31, 2022 at 5:38 PM Theodore Ts'o <tytso@mit.edu> wrote:
> On Sat, Dec 31, 2022 at 04:10:47PM -0500, Dan Cross wrote:
> > >> But in practice, no one has come
> > >> up with a particularly good set of interfaces yet. Ironically,
> > >> BSD-style autoconfig might have been the best yet.
> > >
> > > ??Maybe because it was OS types who knew what the OS needed to discover/report/deliver back from the HW.
> >
> > Perhaps this is what it is, but I think taking a step back and looking
> > at the problem more generally, it's because they're mutated into
> > solving the wrong problem.
>
> More generally, we need to make sure we're all on the same page with
> respect to requirements.  What are we trying to do with respect to
> abstraction?  Do we just want to inform the OS about Port and MMIO
> addresses, interrupts, etc, with the device driver living in the OS?
> Or are are we trying to move the entire device driver plus bus
> configuration/attachment probing into some kind of separate firmware
> layer which is decoupled from the rest of the OS?
>
> How important is portability?  Does this hardware abstraction layer
> and/or firmware need to work with major legacy OS's such as Windows 95
> (still in use by the US Government)?  Windows 10?  NetBSD?  Linux?
> Or just only new research OS's?
>
> How important is performance?  And does it need to support OS's that
> might want to support interesting features, such as say, confidential
> computing?  IOMMU's?  DMA from a storage device directly into memory
> which is accessable over RDMA via Infiniband or iWARP or to GPU
> memory?
>
> Or are we just trying to solve the problem of loading the OS, or maybe
> just initial bootstrap portion of the OS, where performance or support
> for more exotic memory use cases might not be as important?
>
> If we don't agree on exactly what problem that we are trying to solve
> with BSD-style autoconfig, vs UEFI vs ACPI vs the x86 BIOS vs CP/M's
> BIOS, etc.  Otherwise, the risk that we'll end up talking past one
> another, or delivering a solution that we're happy with, but others
> are not, is quite high IMHO.

Spot on, Ted. I agree with everything here.

        - Dan C.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01  3:08     ` Warner Losh
@ 2023-01-01  4:40       ` Dan Cross
  0 siblings, 0 replies; 26+ messages in thread
From: Dan Cross @ 2023-01-01  4:40 UTC (permalink / raw)
  To: Warner Losh; +Cc: Paul Ruizendaal, The Unix Historical Society

On Sat, Dec 31, 2022 at 10:09 PM Warner Losh <imp@bsdimp.com> wrote:
> On Sat, Dec 31, 2022 at 1:03 PM Paul Ruizendaal <pnr@planet.nl> wrote:
>[snip]
> There's been much nasty said about FDT and ACPI, but they do solve real problems: how to enumerate this diversity to the OS in a way that's sane and might not always be as simple as returning a specific number but that requires hardware access to answer even basic questions (because, say, the CPUs were wired this way or that and you have to read those wirings). Linux, even in Linuxboot environments, still uses ACPI, FDT and UEFI to get the job done, and the code there isn't horrific.

This is sort of the issue I have with ACPI+UEFI et al. If we stopped
at what you say (a small nucleus of primordial software that runs once
at the beginning of time intended only to provide essentially static
information to the OS in some relatively sane format, but then dies
and is never consulted again) then either is fine: the ACPI table
formats where this sort of information is encoded are well-defined and
not horrible; ripping through the MADT to find all of your CPUs is
fine.

But that's not all that either did, and the amount of functionality
being shoved into UEFI/ACPI in particular seems to show no sign of
slowing down. I get that having this sort of parallel OS that exposes
functionality in a manner transparent to what we normally consider the
operating system coupled with CPU "features" like SMM means that
vendors can write clever software to hide the fact that you've got a
USB keyboard from an OS that doesn't understand USB; this touches on
the thing Ted mentioned about needing to support old OSes like Windows
95 or whatever. But we're so far beyond compatibility crutches and
into the land of magical black boxes running opaque blobs that do all
sorts of stuff well hidden from the OS, and indeed, the OS has no
control over, by design. THAT's the problem.

> V7 unix for the PDP-11 shipped with maybe 25 drivers total for the whole system, and many of them were quite niche...
>
>> Together this might be a usable Unix BIOS that could have worked in the early 80’s. One could also think of it as a simple hypervisor for only one client. The remaining BBL functionality is not all that different from the content in mch.s on 16-bit Unix (e.g. floating point emulation for CPU’s that don’t have it in hardware). A virtio device is not all that different from the interface presented by e.g. PDP-11 ethernet devices (e.g. DELUA), the MMU abstraction was contemporary.
>
> virtio solves a different problem, though: It's goal is to provide THE interface for mass storage, THE interface for networking, etc so that hypervisor clients can limit their drivers substantially and not have to deal with the thousands of drivers normally needed.

Generally speaking, the hypervisor won't expose hardware devices
directly to the guest. Even with SR-IOV and the like, the HV
necessarily synthesizes, say, PCI config space as seen by the guest
and tightly controls what virtual functions the guest sees, as
anything else allows the guest to usurp the host. This implies that
the HV is providing the guest with virtualized devices anyway, and
once you're doing that the question becomes: what devices to
parameterize the guest with? The HV could emulate things it is fairly
sure the guest already knows about because they're relatively common:
say, an e1000 for a NIC, or AHCI for storage, a 16550 for a console
UART, etc, but what's the relative cost of doing so? It turns out,
most of these devices aren't super great fits for virtualization; they
generate too many exits. Enter virtio, designed for the use case. That
said, hypervisor bypass for virtual devices is a big deal, and not
something that's easily done with virtio. Even offloading virtio
handling to dedicated cores is hard, because there's no easy way for
the guest to generate a doorbell interrupt in the host (kicking a
virtio queue involves a guest exit, which implies some local
processing on the processor running the VCPU: that may be as simple as
kicking off an IPI to another CPU, but you're still exiting, which is
expensive).

> ACPI/FDT just try to make the non-self-describing aspects of the hardware described.

If that were all they did, I think a lot of the complaints would melt away.

>[snip]
> Now, I don't disagree with the org chart args for why they are so large outside of linuxboot, but they do fill a vacuum that would otherwise exist.

The data formats and the general concept of providing that data to the
OS in semi-portable manner, yes. The real problem addressed here is
the decoupling of hardware and systems software at scale; the problem
was simply smaller on the PDP-11 and VAX, but is orders of magnitude
larger now. You need _something_ unless you're in the downright
luxurious position that, say, we're in at Oxide. UEFI+ACPI serve in
that capacity, but it's important to note that that doesn't make them
good. Lots of things that are very useful kind of suck.

        - Dan C.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 20:02   ` Paul Ruizendaal
                       ` (2 preceding siblings ...)
  2023-01-01  3:08     ` Warner Losh
@ 2023-01-01  8:05     ` Jonathan Gray
  3 siblings, 0 replies; 26+ messages in thread
From: Jonathan Gray @ 2023-01-01  8:05 UTC (permalink / raw)
  To: Paul Ruizendaal; +Cc: tuhs

On Sat, Dec 31, 2022 at 09:02:38PM +0100, Paul Ruizendaal wrote:
> 
> 
> > On 31 Dec 2022, at 15:59, Dan Cross <crossd@gmail.com> wrote:
> > 
> > On Fri, Dec 30, 2022 at 1:26 PM Paul Ruizendaal <pnr@planet.nl> wrote:
> >> [snip]
> >> It would seem that the next step for Unix in the area of boot, config and device drivers came with Sun’s OpenBoot in 1988 or so. This also appears to be the first appearance of device trees to describe the hardware to the bios and the kernel. Moreover, it would seem to me that OpenBoot is a spiritual ancestor of the modern Risc-V SBI specification. Maybe by 1988 the IO hardware had become sufficiently complex and/or diverse to warrant a break from tradition?
> >> 
> >> Was there any other notable Unix work on better organising the boot process and the device drivers prior to OpenBoot?
> > 
> > I think that BSD supported autoconfiguration on the VAX well before
> > OpenBoot; the OpenBSD man page says it dates from 4.1 (1981) and was
> > revamped in 4.4.
> 
> That is interesting. Are you referring to this:
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/a/sys/conf
> https://www.tuhs.org/cgi-bin/utree.pl?file=4.1cBSD/usr/man/man8/config.8

"the tuned-up system, with the addition of Robert Elz's auto
configuration code, was released as 4.1BSD in June, 1981"

McKusick, Twenty Years of Berkeley Unix
https://www.oreilly.com/openbook/opensources/book/kirkmck.html

"This update to the fourth distribution of November 1980 provides
support for the VAX 11/750 and for the full interconnect architecture of
the VAX 11/780.  Robert Elz of the University of Melbourne
contributed greatly to this distribution especially in the boot-time
system configuration code"

http://bitsavers.org/bits/UCB_CSRG/4.1_BSD_19810710.zip

4.4 discussed in Torek's
Device Configuration in 4.4BSD
https://www.netbsd.org/docs/kernel/config-torek.ps
https://www.netbsd.org/~mrg/config-torek-fixed.pdf

which references:

A New Framework for Device Support in Berkeley Unix
from Proceedings of the UKUUG, London, Summer 1990.

not online, though Chris sent Warner a copy in 2021
https://minnie.tuhs.org/pipermail/tuhs/2021-April/023756.html
I would also like to read it.  Perhaps it could be added to the archive?

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2022-12-31 22:38       ` Theodore Ts'o
  2022-12-31 22:55         ` Marc Donner
  2023-01-01  3:55         ` Dan Cross
@ 2023-01-01 20:29         ` Paul Ruizendaal
  2023-01-01 21:26           ` G. Branden Robinson
  2 siblings, 1 reply; 26+ messages in thread
From: Paul Ruizendaal @ 2023-01-01 20:29 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: The Eunuchs Hysterical Society


> On 31 Dec 2022, at 23:38, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> If we don't agree on exactly what problem that we are trying to solve
> with BSD-style autoconfig, vs UEFI vs ACPI vs the x86 BIOS vs CP/M's
> BIOS, etc.  Otherwise, the risk that we'll end up talking past one
> another, or delivering a solution that we're happy with, but others
> are not, is quite high IMHO.

Wise words, I was indeed jumping to a conclusion without defining the question. Douglas Adams comes to mind. On the positive side, it did generate a lot of insightful posts that I hope to digest in the coming days.


> On 31 Dec 2022, at 22:10, Dan Cross <crossd@gmail.com> wrote:
> 
> Mothy Roscoe gave a really interesting keynote at OSDI'21:
> https://www.youtube.com/watch?v=36myc8wQhLo

What an interesting keynote! At first he makes the case that OS research is dead (not unlike Rob’s similar observation 20 years before). However, he goes on to point out a specific challenge that he feels is in dire need of research and innovation. In short his case is that a modern SoC is nothing like a VAX, but Linux still treats all hardware like a VAX.

Having watched his keynote I can better place my generic discomfort with the amount of code required outside of the classical OS on a SoC.

====

Below some clarification on the topics that were on my mind:


> That makes figuring out where the host OS image is and getting it loaded
> into memory a real pain in the booty; not to mention on modern systems
> where you’ve got to do memory training as part of initializing DRAM
> controllers and the like.

That was my immediate pain point in doing the D1 SoC port. Unfortunately, the manufacturer only released the DRAM init code as compiler ‘-S’ output and the 1,400 page datasheet does not discuss its registers. Maybe this is a-typical, as I heard in the above keynote that NXP provides 8,000 page datasheets with their SoC’s. Luckily, similar code for ARM chips from this manufacturer was available as C source and I could reverse engineer the assembler file back to about 1,800 lines of C. See https://gitlab.com/pnru/xv6-d1/-/blob/master/boot0/sdram.c 

It does all the expected things (set voltage, switch on the main clock, set up the PLL, calculate delays in terms of clocks, train for the line delays, probe address multiplexing, etc.) by accessing a multitude of registers that appear directly connected to the controller fabric. Yet, at the same time it has only a single entry point (“init_DRAM”) that takes 24 (poorly documented) words of 32 bits to define the desired configuration (with options to specify “default” or “auto”).

Why does the main processor need to run this code? Why is there not a small RV32E CPU inside this controller that runs the 1,800 lines and exposes just the 24 words as registers to the main CPU? Are the square mils really that expensive? Or is this the reason that many SoC’s (but not the D1) have a small side CPU to do this for all devices?

Next, to read from the SD Card, its controller needs to be set up. Luckily here the manufacturer gives out actual C code and there is documentation. However it is another 1,400 lines of code. About half the code goes to initialising the controller hardware, the other half to negotiating with the card to find out the data width and maximum speed. Together with setting up the SoC clock distribution and the console UART I have about 3,500 lines of setup code. And this is just the 1st stage boot loader that has to run from a little bit of SRAM space on the SoC. I have not looked yet at setting up the ethernet, wifi, display, usb, etc., but undoubtedly this is between 1,000 and 2,000 lines each.

This compares to about 7,000 lines for the SysIII kernel.

There is also the question of what goes where. For the SD Card, does a boot level module need to decide or operate the transfer mode? Should this not be in the kernel? Maybe some of this should even be in a daemon? It may seem to be a modern problem, but conceptually these very same questions were at play when doing a compare and contrast between the organisation of networking in 4.2BSD and 8th Edition. The former has nearly everything in the kernel, the latter does a split over device drivers, stream filters and daemons.

This same question also comes up for all the other devices, in particular usb.

====

I’m not quite sure I agree with Rob’s observation on boot ROM’s. While I get his point about organisational causes of bloat, it is not just about the boot ROM. On a RISC-V or ARM SoC the boot sequence is lengthy and the boot rom does little more than bring the system to a safe state and load the first stage boot loader. The full boot process is quite lengthy (e.g. https://riscv.org/wp-content/uploads/2019/12/Summit_bootflow.pdf) and several parts are SoC specific.

One of the things on my mind was that SoC boards change quite quickly: the D1 was last year’s hit with hobbyists, next year it will be something else. Not nice if one has to redo 3,500-10,000 lines of code for each board. Although I am not a fan of Forth, I can see that it is useful when the controller IP blocks of a SoC (or the PCI cards of discrete systems) expose some form of re-targetable program needed to boot it up. The manufacturer BSP code is just not plug & play.

Having watched Mothy Roscoe’s keynote, I realise that I was trying to create a way to make a modern SoC look like a VAX. Maybe valid for my archeology purposes, but not in the general case. On top it did not solve the crappy BSP code issue. Maybe the correct solution for my archeology is to just use the simpler FPGA SoC as a target.

====

> Virtio is solving a very, very different problem: this is limited
> paravirtualization of devices for virtual machines.


Yes, this is where it started, but it seems to get traction as a standardised embedded device interface, especially in automotive (Google for "automotive virtio”). Also in FPGA implementations it seems to get more popular as a device interface e.g. the RVSoc system from the Uni of Tokyo. This one has a very simple & partial HW implementation, but it is enough to support an unmodified Linux.

I don’t think I agree with every observation made on direct virtio implementation (I certainly agree with some) -- but that is maybe off topic for this list. I will observe however that many device controllers of the early-mid-80’s contained some sort of embedded processor, some even on the low cost end of the market -- think of the WD-1002 hard disk controller variant used in the PC-AT. For mini’s this was not unusual either, I already mentioned the M68K on the Unibus DEULA board, the T11 on the Qbus RQDX2 board is another example. Having smart controllers was cost competitive, even in the 80’s.

====

>> - MMU abstraction somewhat similar in idea to that in SYSV/68 [1]
> 
> It's not clear to me what this would buy you; the MMU tends to be
> architecturally defined,

Today, yes, but not in the early 80’s. For the M68K and the Z8000 there were both segmented and paged MMU’s and often custom designs were used (think Onyx C8000, SUN-1).

However, my thinking here was driven by an entirely different thing. I have referred before to the early history of Xenix, for example this chart: http://seefigure1.com/images/xenix/xenix-timeline.jpg (the whole page http://seefigure1.com/2014/04/15/xenixtime.html makes for an interesting read).

For a long time I have wondered why early Xenix did not make the jump to a product that was split between a BIOS and a BDOS part, so that they could sell BDOS-part updates to the total installed base. The BDOS part could even be in some kind of p-code. Considering that they heavily invested in their “revenue bomb” C-compiler at the time, this type of thinking was close to their hart (the Amsterdam Compiler Kit was a similar idea). I am talking ’81-’83 here, thereafter it is clear that their economic interest was to focus on DOS.

There are 3 possibilities:
1. It did not make technical sense
2. It did not make economic sense
3. It did make sense, but it simply did not happen

So, yes, I was conflating a lot of different thoughts into a single solution, without first thinking clearly about the question.

> let's remember that Unix was _very_ success on a variety of
> systems in that era without relying on a BIOS-type abstraction!

For sure, that is undisputed. But could it have been even more successful? Maybe the main reason for "no BIOS attempt" was purely economic: for the companies having to port to each and every machine created customer lock-in, and for the professionals it created an industry of well paying porting & maintenance jobs. The customers were willing to pay for it. Why kill the golden goose?








^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01 20:29         ` Paul Ruizendaal
@ 2023-01-01 21:26           ` G. Branden Robinson
  2023-01-01 21:31             ` Rob Pike
  0 siblings, 1 reply; 26+ messages in thread
From: G. Branden Robinson @ 2023-01-01 21:26 UTC (permalink / raw)
  To: tuhs

[-- Attachment #1: Type: text/plain, Size: 7321 bytes --]

At 2023-01-01T21:29:17+0100, Paul Ruizendaal wrote:
> > Mothy Roscoe gave a really interesting keynote at OSDI'21:
> > https://www.youtube.com/watch?v=36myc8wQhLo
> 
> What an interesting keynote! At first he makes the case that OS
> research is dead (not unlike Rob’s similar observation 20 years
> before).
>
> However, he goes on to point out a specific challenge that he
> feels is in dire need of research and innovation. In short his case is
> that a modern SoC is nothing like a VAX, but Linux still treats all
> hardware like a VAX.

As do C programmers.  https://queue.acm.org/detail.cfm?id=3212479

> > That makes figuring out where the host OS image is and getting it
> > loaded into memory a real pain in the booty; not to mention on
> > modern systems where you’ve got to do memory training as part of
> > initializing DRAM controllers and the like.
> 
> That was my immediate pain point in doing the D1 SoC port.
> Unfortunately, the manufacturer only released the DRAM init code as
> compiler ‘-S’ output and the 1,400 page datasheet does not discuss its
> registers. Maybe this is a-typical, as I heard in the above keynote
> that NXP provides 8,000 page datasheets with their SoC’s. Luckily,
> similar code for ARM chips from this manufacturer was available as C
> source and I could reverse engineer the assembler file back to about
> 1,800 lines of C. See
> https://gitlab.com/pnru/xv6-d1/-/blob/master/boot0/sdram.c 

I don't think it's atypical.  I was pretty annoyed trying to use the
data sheet to program a simple timer chip on the ODROID-C2; the sheet
was simply erroneous about the address of one of the registers you
needed to poke to set up a repeating timer.  When I spoke to people more
experienced with banging on modern devices, I was mostly met with
resignation and rolled shoulders.  I grew up in the hobbyist era; these
days it is _only_ OS nerds who program these devices, and these OS nerds
don't generally handle procurement themselves.  Instead, purchasing
managers do, and those people don't have to face the pain.

Data sheets are only as good as they need to be to move the product,
which means they don't need to be good at all, since the people who pay
for them look only at the advertised feature list and the price.

> It does all the expected things (set voltage, switch on the main
> clock, set up the PLL, calculate delays in terms of clocks, train for
> the line delays, probe address multiplexing, etc.) by accessing a
> multitude of registers that appear directly connected to the
> controller fabric. Yet, at the same time it has only a single entry
> point (“init_DRAM”) that takes 24 (poorly documented) words of 32 bits
> to define the desired configuration (with options to specify “default”
> or “auto”).
> 
> Why does the main processor need to run this code? Why is there not a
> small RV32E CPU inside this controller that runs the 1,800 lines and
> exposes just the 24 words as registers to the main CPU? Are the square
> mils really that expensive? Or is this the reason that many SoC’s (but
> not the D1) have a small side CPU to do this for all devices?

I can't speak to the economic arguments but it seems wasteful to me to
have an auxiliary CPU that is--or _should_--be used only to bring up the
board.  The other problem I have is more ominous.

"Ah, BMC.  Now every computer comes with an extra full-fledged computer!
The main computer is for your use, and the other computer is for the use
of the attacker." — Russ Allbery

But the real problem is with the software these auxiliary processors
run.  My first encounter with auxiliary processors was with the TRS-80
Model 16 line in the early 1980s, which was also available as an upgrade
option from the Model II.  The II ran a Zilog Z80.  The Model 16 upgrade
added a 68000 board which became the main CPU, and the Z80 was relegated
to I/O management.  This turned out to work really well.  I gather that
big IBM iron was doing this sort of thing decades earlier.  That at
least is an honorable and useful application of the additional core,
instead of smuggling your keystrokes and network packets out to
interested intelligence agencies and harvesters for targeted
advertising.

[...]
> One of the things on my mind was that SoC boards change quite quickly:
> the D1 was last year’s hit with hobbyists, next year it will be
> something else. Not nice if one has to redo 3,500-10,000 lines of code
> for each board. Although I am not a fan of Forth, I can see that it is
> useful when the controller IP blocks of a SoC (or the PCI cards of
> discrete systems) expose some form of re-targetable program needed to
> boot it up. The manufacturer BSP code is just not plug & play.

Yes.  I too think Forth is a little weird but I appreciate its
power/memory footprint ratio.  I really admire OpenFirmware and was
intensely disappointed when the RISC-V community didn't take the
opportunity to resurrect it.  Forth code seems a lot more auditable to
me than machine language blobs.

> Maybe the correct solution for my archeology is to just use the
> simpler FPGA SoC as a target.

Maybe it's the correct solution for anyone who cares about a verified
boot process, too.  No one who sells silicon seems to.

> For a long time I have wondered why early Xenix did not make the jump
> to a product that was split between a BIOS and a BDOS part, so that
> they could sell BDOS-part updates to the total installed base. The
> BDOS part could even be in some kind of p-code. Considering that they
> heavily invested in their “revenue bomb” C-compiler at the time, this
> type of thinking was close to their hart

You _really have_ been writing for RISC-V lately... ;-)

> (the Amsterdam Compiler Kit was a similar idea). I am talking ’81-’83
> here, thereafter it is clear that their economic interest was to focus
> on DOS.
> 
> There are 3 possibilities:
> 1. It did not make technical sense
> 2. It did not make economic sense
> 3. It did make sense, but it simply did not happen
> 
> So, yes, I was conflating a lot of different thoughts into a single
> solution, without first thinking clearly about the question.

Someone is bound to suggest that p-code was, or still is, detrimental to
boot times.  But cores are so much faster than memory buses, and JIT
compilers so far advanced over what was available in the 1980s, that I
wonder if that is simply a myth today for any implementation that didn't
go out of its way to be slow.

> For sure, that is undisputed. But could it have been even more
> successful? Maybe the main reason for "no BIOS attempt" was purely
> economic: for the companies having to port to each and every machine
> created customer lock-in, and for the professionals it created an
> industry of well paying porting & maintenance jobs. The customers were
> willing to pay for it. Why kill the golden goose?

Never compete on margins when you can extract monopoly rents, leverage
asymmetric information, impose negative externalities, and defraud your
customer or the public generally.  Everybody who stops learning
economics after their first micro class is a sheep waiting your shears.

Regards,
Branden

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

* [TUHS] Re: Porting the SysIII kernel: boot, config & device drivers
  2023-01-01 21:26           ` G. Branden Robinson
@ 2023-01-01 21:31             ` Rob Pike
  0 siblings, 0 replies; 26+ messages in thread
From: Rob Pike @ 2023-01-01 21:31 UTC (permalink / raw)
  To: G. Branden Robinson; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 8048 bytes --]

Seems like a good time to pull out the "wheel of reincarnation" paper.

http://cva.stanford.edu/classes/cs99s/papers/myer-sutherland-design-of-display-processors.pdf

I think about it a lot when I ponder the state of graphics these days. The
wheel does seem finally to have rusted in place, although I'm not sure I'd
choose this exact angle of rotation.

-rob


On Mon, Jan 2, 2023 at 8:27 AM G. Branden Robinson <
g.branden.robinson@gmail.com> wrote:

> At 2023-01-01T21:29:17+0100, Paul Ruizendaal wrote:
> > > Mothy Roscoe gave a really interesting keynote at OSDI'21:
> > > https://www.youtube.com/watch?v=36myc8wQhLo
> >
> > What an interesting keynote! At first he makes the case that OS
> > research is dead (not unlike Rob’s similar observation 20 years
> > before).
> >
> > However, he goes on to point out a specific challenge that he
> > feels is in dire need of research and innovation. In short his case is
> > that a modern SoC is nothing like a VAX, but Linux still treats all
> > hardware like a VAX.
>
> As do C programmers.  https://queue.acm.org/detail.cfm?id=3212479
>
> > > That makes figuring out where the host OS image is and getting it
> > > loaded into memory a real pain in the booty; not to mention on
> > > modern systems where you’ve got to do memory training as part of
> > > initializing DRAM controllers and the like.
> >
> > That was my immediate pain point in doing the D1 SoC port.
> > Unfortunately, the manufacturer only released the DRAM init code as
> > compiler ‘-S’ output and the 1,400 page datasheet does not discuss its
> > registers. Maybe this is a-typical, as I heard in the above keynote
> > that NXP provides 8,000 page datasheets with their SoC’s. Luckily,
> > similar code for ARM chips from this manufacturer was available as C
> > source and I could reverse engineer the assembler file back to about
> > 1,800 lines of C. See
> > https://gitlab.com/pnru/xv6-d1/-/blob/master/boot0/sdram.c
>
> I don't think it's atypical.  I was pretty annoyed trying to use the
> data sheet to program a simple timer chip on the ODROID-C2; the sheet
> was simply erroneous about the address of one of the registers you
> needed to poke to set up a repeating timer.  When I spoke to people more
> experienced with banging on modern devices, I was mostly met with
> resignation and rolled shoulders.  I grew up in the hobbyist era; these
> days it is _only_ OS nerds who program these devices, and these OS nerds
> don't generally handle procurement themselves.  Instead, purchasing
> managers do, and those people don't have to face the pain.
>
> Data sheets are only as good as they need to be to move the product,
> which means they don't need to be good at all, since the people who pay
> for them look only at the advertised feature list and the price.
>
> > It does all the expected things (set voltage, switch on the main
> > clock, set up the PLL, calculate delays in terms of clocks, train for
> > the line delays, probe address multiplexing, etc.) by accessing a
> > multitude of registers that appear directly connected to the
> > controller fabric. Yet, at the same time it has only a single entry
> > point (“init_DRAM”) that takes 24 (poorly documented) words of 32 bits
> > to define the desired configuration (with options to specify “default”
> > or “auto”).
> >
> > Why does the main processor need to run this code? Why is there not a
> > small RV32E CPU inside this controller that runs the 1,800 lines and
> > exposes just the 24 words as registers to the main CPU? Are the square
> > mils really that expensive? Or is this the reason that many SoC’s (but
> > not the D1) have a small side CPU to do this for all devices?
>
> I can't speak to the economic arguments but it seems wasteful to me to
> have an auxiliary CPU that is--or _should_--be used only to bring up the
> board.  The other problem I have is more ominous.
>
> "Ah, BMC.  Now every computer comes with an extra full-fledged computer!
> The main computer is for your use, and the other computer is for the use
> of the attacker." — Russ Allbery
>
> But the real problem is with the software these auxiliary processors
> run.  My first encounter with auxiliary processors was with the TRS-80
> Model 16 line in the early 1980s, which was also available as an upgrade
> option from the Model II.  The II ran a Zilog Z80.  The Model 16 upgrade
> added a 68000 board which became the main CPU, and the Z80 was relegated
> to I/O management.  This turned out to work really well.  I gather that
> big IBM iron was doing this sort of thing decades earlier.  That at
> least is an honorable and useful application of the additional core,
> instead of smuggling your keystrokes and network packets out to
> interested intelligence agencies and harvesters for targeted
> advertising.
>
> [...]
> > One of the things on my mind was that SoC boards change quite quickly:
> > the D1 was last year’s hit with hobbyists, next year it will be
> > something else. Not nice if one has to redo 3,500-10,000 lines of code
> > for each board. Although I am not a fan of Forth, I can see that it is
> > useful when the controller IP blocks of a SoC (or the PCI cards of
> > discrete systems) expose some form of re-targetable program needed to
> > boot it up. The manufacturer BSP code is just not plug & play.
>
> Yes.  I too think Forth is a little weird but I appreciate its
> power/memory footprint ratio.  I really admire OpenFirmware and was
> intensely disappointed when the RISC-V community didn't take the
> opportunity to resurrect it.  Forth code seems a lot more auditable to
> me than machine language blobs.
>
> > Maybe the correct solution for my archeology is to just use the
> > simpler FPGA SoC as a target.
>
> Maybe it's the correct solution for anyone who cares about a verified
> boot process, too.  No one who sells silicon seems to.
>
> > For a long time I have wondered why early Xenix did not make the jump
> > to a product that was split between a BIOS and a BDOS part, so that
> > they could sell BDOS-part updates to the total installed base. The
> > BDOS part could even be in some kind of p-code. Considering that they
> > heavily invested in their “revenue bomb” C-compiler at the time, this
> > type of thinking was close to their hart
>
> You _really have_ been writing for RISC-V lately... ;-)
>
> > (the Amsterdam Compiler Kit was a similar idea). I am talking ’81-’83
> > here, thereafter it is clear that their economic interest was to focus
> > on DOS.
> >
> > There are 3 possibilities:
> > 1. It did not make technical sense
> > 2. It did not make economic sense
> > 3. It did make sense, but it simply did not happen
> >
> > So, yes, I was conflating a lot of different thoughts into a single
> > solution, without first thinking clearly about the question.
>
> Someone is bound to suggest that p-code was, or still is, detrimental to
> boot times.  But cores are so much faster than memory buses, and JIT
> compilers so far advanced over what was available in the 1980s, that I
> wonder if that is simply a myth today for any implementation that didn't
> go out of its way to be slow.
>
> > For sure, that is undisputed. But could it have been even more
> > successful? Maybe the main reason for "no BIOS attempt" was purely
> > economic: for the companies having to port to each and every machine
> > created customer lock-in, and for the professionals it created an
> > industry of well paying porting & maintenance jobs. The customers were
> > willing to pay for it. Why kill the golden goose?
>
> Never compete on margins when you can extract monopoly rents, leverage
> asymmetric information, impose negative externalities, and defraud your
> customer or the public generally.  Everybody who stops learning
> economics after their first micro class is a sheep waiting your shears.
>
> Regards,
> Branden
>

[-- Attachment #2: Type: text/html, Size: 9930 bytes --]

^ permalink raw reply	[flat|nested] 26+ messages in thread

end of thread, other threads:[~2023-01-01 21:32 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-12-30 18:25 [TUHS] Porting the SysIII kernel: boot, config & device drivers Paul Ruizendaal
2022-12-30 18:56 ` [TUHS] " Steve Nickolas
2022-12-31 14:59 ` Dan Cross
2022-12-31 19:08   ` Clem Cole
2022-12-31 21:10     ` Dan Cross
2022-12-31 21:39       ` Clem Cole
2022-12-31 21:52         ` Dan Cross
2022-12-31 23:25         ` Dave Horsfall
2023-01-01  1:02           ` Rob Pike
2023-01-01  1:16             ` George Michaelson
2023-01-01  1:40               ` Larry McVoy
2023-01-01  2:29                 ` Warner Losh
2023-01-01  1:24             ` Larry McVoy
2022-12-31 22:38       ` Theodore Ts'o
2022-12-31 22:55         ` Marc Donner
2023-01-01  3:55         ` Dan Cross
2023-01-01 20:29         ` Paul Ruizendaal
2023-01-01 21:26           ` G. Branden Robinson
2023-01-01 21:31             ` Rob Pike
2022-12-31 21:11     ` Paul Ruizendaal
2022-12-31 20:02   ` Paul Ruizendaal
2022-12-31 21:04     ` Warner Losh
2022-12-31 21:41     ` Dan Cross
2023-01-01  3:08     ` Warner Losh
2023-01-01  4:40       ` Dan Cross
2023-01-01  8:05     ` Jonathan Gray

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).