[TUHS] Re: Porting the SysIII kernel: boot, config & device drivers

[Paul Ruizendaal <pnr@planet.nl>]

> On 31 Dec 2022, at 23:38, Theodore Ts'o <tytso@mit.edu> wrote:
> 
> If we don't agree on exactly what problem that we are trying to solve
> with BSD-style autoconfig, vs UEFI vs ACPI vs the x86 BIOS vs CP/M's
> BIOS, etc.  Otherwise, the risk that we'll end up talking past one
> another, or delivering a solution that we're happy with, but others
> are not, is quite high IMHO.

Wise words, I was indeed jumping to a conclusion without defining the question. Douglas Adams comes to mind. On the positive side, it did generate a lot of insightful posts that I hope to digest in the coming days.


> On 31 Dec 2022, at 22:10, Dan Cross <crossd@gmail.com> wrote:
> 
> Mothy Roscoe gave a really interesting keynote at OSDI'21:
> https://www.youtube.com/watch?v=36myc8wQhLo

What an interesting keynote! At first he makes the case that OS research is dead (not unlike Rob’s similar observation 20 years before). However, he goes on to point out a specific challenge that he feels is in dire need of research and innovation. In short his case is that a modern SoC is nothing like a VAX, but Linux still treats all hardware like a VAX.

Having watched his keynote I can better place my generic discomfort with the amount of code required outside of the classical OS on a SoC.

====

Below some clarification on the topics that were on my mind:


> That makes figuring out where the host OS image is and getting it loaded
> into memory a real pain in the booty; not to mention on modern systems
> where you’ve got to do memory training as part of initializing DRAM
> controllers and the like.

That was my immediate pain point in doing the D1 SoC port. Unfortunately, the manufacturer only released the DRAM init code as compiler ‘-S’ output and the 1,400 page datasheet does not discuss its registers. Maybe this is a-typical, as I heard in the above keynote that NXP provides 8,000 page datasheets with their SoC’s. Luckily, similar code for ARM chips from this manufacturer was available as C source and I could reverse engineer the assembler file back to about 1,800 lines of C. See https://gitlab.com/pnru/xv6-d1/-/blob/master/boot0/sdram.c 

It does all the expected things (set voltage, switch on the main clock, set up the PLL, calculate delays in terms of clocks, train for the line delays, probe address multiplexing, etc.) by accessing a multitude of registers that appear directly connected to the controller fabric. Yet, at the same time it has only a single entry point (“init_DRAM”) that takes 24 (poorly documented) words of 32 bits to define the desired configuration (with options to specify “default” or “auto”).

Why does the main processor need to run this code? Why is there not a small RV32E CPU inside this controller that runs the 1,800 lines and exposes just the 24 words as registers to the main CPU? Are the square mils really that expensive? Or is this the reason that many SoC’s (but not the D1) have a small side CPU to do this for all devices?

Next, to read from the SD Card, its controller needs to be set up. Luckily here the manufacturer gives out actual C code and there is documentation. However it is another 1,400 lines of code. About half the code goes to initialising the controller hardware, the other half to negotiating with the card to find out the data width and maximum speed. Together with setting up the SoC clock distribution and the console UART I have about 3,500 lines of setup code. And this is just the 1st stage boot loader that has to run from a little bit of SRAM space on the SoC. I have not looked yet at setting up the ethernet, wifi, display, usb, etc., but undoubtedly this is between 1,000 and 2,000 lines each.

This compares to about 7,000 lines for the SysIII kernel.

There is also the question of what goes where. For the SD Card, does a boot level module need to decide or operate the transfer mode? Should this not be in the kernel? Maybe some of this should even be in a daemon? It may seem to be a modern problem, but conceptually these very same questions were at play when doing a compare and contrast between the organisation of networking in 4.2BSD and 8th Edition. The former has nearly everything in the kernel, the latter does a split over device drivers, stream filters and daemons.

This same question also comes up for all the other devices, in particular usb.

====

I’m not quite sure I agree with Rob’s observation on boot ROM’s. While I get his point about organisational causes of bloat, it is not just about the boot ROM. On a RISC-V or ARM SoC the boot sequence is lengthy and the boot rom does little more than bring the system to a safe state and load the first stage boot loader. The full boot process is quite lengthy (e.g. https://riscv.org/wp-content/uploads/2019/12/Summit_bootflow.pdf) and several parts are SoC specific.

One of the things on my mind was that SoC boards change quite quickly: the D1 was last year’s hit with hobbyists, next year it will be something else. Not nice if one has to redo 3,500-10,000 lines of code for each board. Although I am not a fan of Forth, I can see that it is useful when the controller IP blocks of a SoC (or the PCI cards of discrete systems) expose some form of re-targetable program needed to boot it up. The manufacturer BSP code is just not plug & play.

Having watched Mothy Roscoe’s keynote, I realise that I was trying to create a way to make a modern SoC look like a VAX. Maybe valid for my archeology purposes, but not in the general case. On top it did not solve the crappy BSP code issue. Maybe the correct solution for my archeology is to just use the simpler FPGA SoC as a target.

====

> Virtio is solving a very, very different problem: this is limited
> paravirtualization of devices for virtual machines.


Yes, this is where it started, but it seems to get traction as a standardised embedded device interface, especially in automotive (Google for "automotive virtio”). Also in FPGA implementations it seems to get more popular as a device interface e.g. the RVSoc system from the Uni of Tokyo. This one has a very simple & partial HW implementation, but it is enough to support an unmodified Linux.

I don’t think I agree with every observation made on direct virtio implementation (I certainly agree with some) -- but that is maybe off topic for this list. I will observe however that many device controllers of the early-mid-80’s contained some sort of embedded processor, some even on the low cost end of the market -- think of the WD-1002 hard disk controller variant used in the PC-AT. For mini’s this was not unusual either, I already mentioned the M68K on the Unibus DEULA board, the T11 on the Qbus RQDX2 board is another example. Having smart controllers was cost competitive, even in the 80’s.

====

>> - MMU abstraction somewhat similar in idea to that in SYSV/68 [1]
> 
> It's not clear to me what this would buy you; the MMU tends to be
> architecturally defined,

Today, yes, but not in the early 80’s. For the M68K and the Z8000 there were both segmented and paged MMU’s and often custom designs were used (think Onyx C8000, SUN-1).

However, my thinking here was driven by an entirely different thing. I have referred before to the early history of Xenix, for example this chart: http://seefigure1.com/images/xenix/xenix-timeline.jpg (the whole page http://seefigure1.com/2014/04/15/xenixtime.html makes for an interesting read).

For a long time I have wondered why early Xenix did not make the jump to a product that was split between a BIOS and a BDOS part, so that they could sell BDOS-part updates to the total installed base. The BDOS part could even be in some kind of p-code. Considering that they heavily invested in their “revenue bomb” C-compiler at the time, this type of thinking was close to their hart (the Amsterdam Compiler Kit was a similar idea). I am talking ’81-’83 here, thereafter it is clear that their economic interest was to focus on DOS.

There are 3 possibilities:
1. It did not make technical sense
2. It did not make economic sense
3. It did make sense, but it simply did not happen

So, yes, I was conflating a lot of different thoughts into a single solution, without first thinking clearly about the question.

> let's remember that Unix was _very_ success on a variety of
> systems in that era without relying on a BIOS-type abstraction!

For sure, that is undisputed. But could it have been even more successful? Maybe the main reason for "no BIOS attempt" was purely economic: for the companies having to port to each and every machine created customer lock-in, and for the professionals it created an industry of well paying porting & maintenance jobs. The customers were willing to pay for it. Why kill the golden goose?