[TUHS] Re: kernel boots kernel in 1977

[TUHS] Re: kernel boots kernel in 1977

[Noel Chiappa]

    > From: Ron Minnich

    > Ed got tired of watching the bootstrap slowness

This may be a dumb question (in which case, my apologies), but which part of
booting a PDP-11 UNIX was slow? And thus, which parts did he bypass in a
'quick reboot'? (I'm having a bit of a hard time working it out.) I can think
of the following stages as potentially being 'slow':

1 - reading in the kernel image from disk
2 - sizing/clearing main memory
3 - loading running /etc/init
3A - creating all the 'loqin's
3B - starting all the daemons/etc with /etc/rc

(There's obviously a conflict between 2 and 3*; one can't avoid 3* if one
does 2.)

Which ones did he want to avoid?

Avoiding 3* puts some limitations on the system, obviously, because it means
one has to keep running processes around; the process table has to have the
same format (and be in the same location - or it has to be moved before the
new system is started). (Technically, I guess one would have to save the
inode and file tables, too; there isn't enough saved data in the kernel to
re-open files, plus there are file read/write ocation pointers, etc.)

One could sort of do 2 also, if one were prepared to 1) swap all processes
out to secondary storage before 'rebooting', and ii) saving the process table.

    > But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel"
    > or was there something before?

Are you talking about for UNIX, or across all operating systems? I don't have
that broad a memory any more, to know - did TWENEX have something like that?

     Noel

[TUHS] Re: kernel boots kernel in 1977

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]

It's been too long. Plus, for all I know, it may have been a "wouldn't this
be cool" project that did not work out. I don't recall that we ended up
using it much. But that was almost 50 years ago, so your guess is as good
as mine.

It was for sure a cute hack.

On Thu, Sep 19, 2024 at 12:10 PM Noel Chiappa <jnc@mercury.lcs.mit.edu>
wrote:

>     > From: Ron Minnich
>
>     > Ed got tired of watching the bootstrap slowness
>
> This may be a dumb question (in which case, my apologies), but which part
> of
> booting a PDP-11 UNIX was slow? And thus, which parts did he bypass in a
> 'quick reboot'? (I'm having a bit of a hard time working it out.) I can
> think
> of the following stages as potentially being 'slow':
>
> 1 - reading in the kernel image from disk
> 2 - sizing/clearing main memory
> 3 - loading running /etc/init
> 3A - creating all the 'loqin's
> 3B - starting all the daemons/etc with /etc/rc
>
> (There's obviously a conflict between 2 and 3*; one can't avoid 3* if one
> does 2.)
>
> Which ones did he want to avoid?
>
> Avoiding 3* puts some limitations on the system, obviously, because it
> means
> one has to keep running processes around; the process table has to have the
> same format (and be in the same location - or it has to be moved before the
> new system is started). (Technically, I guess one would have to save the
> inode and file tables, too; there isn't enough saved data in the kernel to
> re-open files, plus there are file read/write ocation pointers, etc.)
>
> One could sort of do 2 also, if one were prepared to 1) swap all processes
> out to secondary storage before 'rebooting', and ii) saving the process
> table.
>
>     > But I'm wondering: is Ed's work in 1977 the first "kernel boots
> kernel"
>     > or was there something before?
>
> Are you talking about for UNIX, or across all operating systems? I don't
> have
> that broad a memory any more, to know - did TWENEX have something like
> that?
>
>      Noel
>

[-- Attachment #2: Type: text/html, Size: 2499 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Brian Walden]

I can see an immedaite advantage in
        1 - reading in the kernel image from disk

As late as August of 1983, as that's the last time I had to do it, there were still systems in operation that you had to manually input the boot loader by hand using front panel switches.

See https://gunkies.org/wiki/PDP-11_Bootstrap_Loader

-Brian

> ron minnich rminnich at gmail.com
> Fri Sep 20 05:42:45 AEST 2024
>
>
> It's been too long. Plus, for all I know, it may have been a "wouldn't this
> be cool" project that did not work out. I don't recall that we ended up
> using it much. But that was almost 50 years ago, so your guess is as good
> as mine.
>
> It was for sure a cute hack.
>
> On Thu, Sep 19, 2024 at 12:10b\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0

[TUHS] Re: kernel boots kernel in 1977

[Steffen Nurpmeso]

Bakul Shah wrote in
 <D4467141-6023-4166-B318-33D15D314F5D@iitbombay.org>:
 |> On Sep 19, 2024, at 5:20 PM, Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
 |> Bakul Shah via TUHS wrote in
 |> <3B4CF41C-3D88-471D-B5E7-6F06C772F5E7@iitbombay.org>:
 ...
 |>|It is not so simple these days as the hardware is much more
 |>|complex and often requires vendor provided firmware assist
 |>|to properly initialize the system before control gets passed
 |>|to an OS kernel but no change in the protection model is
 |>|required for a kernel to kernel handoff.
 |> 
 |> It seems to me it is worse.  The Linux driver of my WLAN chip
 |> would (i have forgotten the condition, but anyhow) leave the chip
 |> in some false state, and it could not be overcome except by
 ...
 |> I do not know whether the flag is still needed...
 |> 
 |>  KEXEC_ARGS="--append=\"rtw88_pci.disable_aspm=1 rc.hostname=kent\""
 |
 |I don't see how "it is worse".

Ok, yes, binary blobs of some firmware may make the pager out of
you, which is definetely worse than having "free" drivers on top
of them which fail to perform proper initialization.

(..i am not a kernel hacker, but the fix could have been
44492e70adc8086c42d3745d21d591657a427f04, (rtw88: pci: Power cycle
device during shutdown, 2020-09-29), "platform firmware may not
properly power manage the device during shutdown[.] putting the
device to D3 can workaround the issue".

  +       pci_set_power_state(pdev, PCI_D3hot);

And so many quirks, everywhere.)

 --End of <D4467141-6023-4166-B318-33D15D314F5D@iitbombay.org>

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: kernel boots kernel in 1977

[Bakul Shah via TUHS]

> On Sep 19, 2024, at 5:20 PM, Steffen Nurpmeso <steffen@sdaoden.eu> wrote:
> 
> Bakul Shah via TUHS wrote in
> <3B4CF41C-3D88-471D-B5E7-6F06C772F5E7@iitbombay.org>:
> |Forget "rings". Unix needs only two states from hardware.
> |Forget virtualization for now as well.
> |
> |The kernel runs in the supervisor state, the user code in the
> |unprivileged state where it can't execute or see certain
> |instructions or processor state or peripherals and must make
> |system calls for controlled access to the same.
> |
> |In addition Unix has the "root" user that can access much more
> |but it too must go through the system call interface.
> |
> |A "warm reboot" system call would simply arrange things so that
> |the new kernel image is copied in the right place and control
> |will eventually pass to it.
> |
> |It is not so simple these days as the hardware is much more
> |complex and often requires vendor provided firmware assist
> |to properly initialize the system before control gets passed
> |to an OS kernel but no change in the protection model is
> |required for a kernel to kernel handoff.
> 
> It seems to me it is worse.  The Linux driver of my WLAN chip
> would (i have forgotten the condition, but anyhow) leave the chip
> in some false state, and it could not be overcome except by
> booting into the (minimzed as much as possible: 31.4G) by-default
> installed Windows, up to the login screen, by then it would have
> been fixed.  This was with the RTW88 driver, it was ok iirc in
> 4.19, which had another one.  Somewhere i have found someone
> saying one has to pass an argument, i asked for documentation
> i think even via bugzilla, but nothing ever happened.  But it
> worked (and the driver became better, and i think firmware also,
> both "a bit"), and now it is luckily stable again for some years.
> I do not know whether the flag is still needed...
> 
>  KEXEC_ARGS="--append=\"rtw88_pci.disable_aspm=1 rc.hostname=kent\""

I don't see how "it is worse".

[TUHS] Re: kernel boots kernel in 1977

[Steffen Nurpmeso]

Bakul Shah via TUHS wrote in
 <3B4CF41C-3D88-471D-B5E7-6F06C772F5E7@iitbombay.org>:
 |Forget "rings". Unix needs only two states from hardware.
 |Forget virtualization for now as well.
 |
 |The kernel runs in the supervisor state, the user code in the
 |unprivileged state where it can't execute or see certain
 |instructions or processor state or peripherals and must make
 |system calls for controlled access to the same.
 |
 |In addition Unix has the "root" user that can access much more
 |but it too must go through the system call interface.
 |
 |A "warm reboot" system call would simply arrange things so that
 |the new kernel image is copied in the right place and control
 |will eventually pass to it.
 |
 |It is not so simple these days as the hardware is much more
 |complex and often requires vendor provided firmware assist
 |to properly initialize the system before control gets passed
 |to an OS kernel but no change in the protection model is
 |required for a kernel to kernel handoff.

It seems to me it is worse.  The Linux driver of my WLAN chip
would (i have forgotten the condition, but anyhow) leave the chip
in some false state, and it could not be overcome except by
booting into the (minimzed as much as possible: 31.4G) by-default
installed Windows, up to the login screen, by then it would have
been fixed.  This was with the RTW88 driver, it was ok iirc in
4.19, which had another one.  Somewhere i have found someone
saying one has to pass an argument, i asked for documentation
i think even via bugzilla, but nothing ever happened.  But it
worked (and the driver became better, and i think firmware also,
both "a bit"), and now it is luckily stable again for some years.
I do not know whether the flag is still needed...

  KEXEC_ARGS="--append=\"rtw88_pci.disable_aspm=1 rc.hostname=kent\""

--steffen
|
|Der Kragenbaer,                The moon bear,
|der holt sich munter           he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

[TUHS] Re: kernel boots kernel in 1977

[Dan Cross]

On Wed, Sep 18, 2024 at 9:54 PM Bakul Shah <bakul@iitbombay.org> wrote:
> > On Sep 18, 2024, at 5:53 PM, Dan Cross <crossd@gmail.com> wrote:
> > On Wed, Sep 18, 2024 at 8:05 PM Bakul Shah via TUHS <tuhs@tuhs.org> wrote:
> >> Can you not avoid resetting the machine? This can be treated almost as sleep in the old kernel, wakeup in the new one! You do have to reset devices individually (which may not always work if it requires assistance from some undocumented firmware).
> >
> > Perhaps this is what you mean when you mention assistance from
> > firmware, Bakul, but it may be useful to consider that _many_ devices
> > are touched by e.g. a BIOS or UEFI or whatever well before the OS is
> > even loaded.
>
> Right but presumably the old kernel leaves them in a good enough state.

I suspect this is one of the thornier parts of the whole problem. In
some sense, kexec is similar to live migration of a VM: it's certainly
possible to do, but in particular devices have to be quiesced and in a
state where they are ready to migrate; outstanding IOs may cause
problems with synchronization between the source and destination.
Similarly, if the outgoing kernel in a kexec cannot adequately ensure
that device state is going to be (at a minimum) discoverable in the
incoming kernel, you're going to have a bad time. If there's an
outstanding DMA request? Well, good luck, but you're likely going to
have a bad day....

> > If one steps back and considers the utility of a BIOS/UEFI (and I
> > often lump these into the same category), there are three principal
> > reasons for it: 1) back in the bad old days, we could offload common
> > IO functions into code stored on a ROM, freeing up precious RAM for
> > programs. 2) firmware provides a layer of indirection between the
> > system and the host software, allowing both to vary while continuing
> > to work with newer versions of the other. And finally 3) firmware
> > facilitates bootstrapping the system by providing the host some way to
> > access devices and locate and load an OS image, er, before the OS
> > image is loaded. SOMETHING has to get enough code loaded from
> > somewhere to start the system; often times that's firmware.
>
> The new OS image is already in memory but may need to be copied to
> the right place. The devices were already working (but may need to
> have their interrupts disabled and any DMA stopped etc.).

Yes, sorry, I was trying to explain why firmware is in the loop for
those who may not be familiar.

> > Anyway, the last two suggest that device state can be arbitrarily
> > munged before the OS takes over, and an actual reset at the device
> > level might wipe out some state the OS depends on. Consider, for
> > example, programming PCI BARs; on a "modern" x86-64 system with UEFI,
> > this is done by firmware in the PEI layer, and the OS may expect that
> > to already be set up by the time it is probing buses. An actual
> > honest-to-goodness reset will probably wipe the BARs, requiring the
> > host OS to program them (ironically, many OSes are already equipped to
> > do so, as they have to handle these cases for e.g. PCI hotplug events,
> > though many don't do it in the "ordinary" discovery and initialization
> > phase of boot).
>
> All that is done on powerup.

That's true, but it's non-trivial, and done by opaque firmware that
one has no control over; in particular, it's hard to get the firmware
to cooperate in the kexec protocol.

> > I suppose the point is that a reset is great because it really does
> > wipe out state, but it may also be a bummer because, well, it really
> > does wipe out state. :-)
>
> :-) I was speculating that kernel to kernel warmboot should be doable.

Oh sorry; I think I misunderstood that and thought you were asking,
"why can't you reset the machine?" Apologies there; my bad. I
absolutely agree that it is doable, and that we have several existence
proofs showing just that.

        - Dan C.

[TUHS] Re: kernel boots kernel in 1977

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 8962 bytes --]

On Thu, Sep 19, 2024, 3:51 PM ron minnich <rminnich@gmail.com> wrote:

> to reiterate: you can avoid resetting the machine, and for all the x
> million systems in data centers around the world, we do avoid resetting the
> machine.
>
> But that comes with its own set of issues, including kernel version x not
> being able to boot kernel version y (very common with linux, no problem on
> plan 9); and hardware not behaving well, since few people write drivers
> that properly reset hardware; or the hardware can't be cleaned up absent a
> reset (most common problem areas are NICs and graphics). Very few linux
> drivers can properly shut down hardware for a kexec. The IOMMU and MSIx
> added a whole new world of fun. Sometimes it feels like it works by
> accident.
>
> Also recall that side channels across a kexec are an issue that has to be
> considered. At Google we've considered them by, e.g., turning on "zero on
> free" and "zero on alloc" in the kernel that will kexec, only using a small
> amount of memory in the first kernel (32GiB is small! Ha!), among other
> things. But since DRAM SPD and Voltage regulator module FLASH provide
> places to hide things, it's getting messy.
>
> So it's not as simple as "reset bad, not reset good". Hardware is poorly
> designed, or the drivers are poorly written, and absent a reset, the
> kexec'ed kernel may fail to boot -- lockup is common, panic is common. That
> said, no system I know of implements kexec with a reset in the middle.
>
> Short history: in the Linux world, kernel boots kernel was done, in 1999,
> by LOBOS (me) and Eric Hendriks (Two Kernel Monte), and Alpha Power
> (DBLX). Werner Almesberger did his own thing ca. 2000 called (iirc)
> bootimg. Eric Biederman looked at LOBOS, did not like it, and wrote kexec,
> I believe around 2001. Plan 9 got kernel boots kernel around that time. As
> usual, the Plan 9 implementation was the most compact and cleanest. This
> paper https://ieeexplore.ieee.org/document/1392643 compares them.
>
> The AlphaPower DBLX code was lost when the company went under. They made a
> heroic effort to get it to sourceforge but things happened too fast. DBLX
> means "direct boot linux" -- the acronym reads better.
>
> The first kexec was a very general interface, with a Himalayan learning
> curve. At some point an Intel engineer found kexec confusing and wrote an
> entirely new type of kexec, with a different API, that many people found
> easier.
>

The kexec I'm ising is like that. "Load the memory you want and give us a
start address." is all the instructions you get. God speed. Best of luck.
Have fun storming the castle.

I had to read a ton of code to find the details. Then I needed to set it up
like FreeBSD's regular loaders do. And once I guessed wrong about 200
times, i was up and limping. I'd definitely wouldn't call kexec easy to
learn or code to...

And it was a blast...

Warner

so kexec has been around for 20 years, and we're still getting the hang of
> it, and there are still people who claim that it will never fully work.
>
> Anyway, we're far afield of the original question, but it was very
> interesting to read how far back the idea goes! PDP-7, who knew?
>
> p.s. as to an unrelated discussion: kernels have been self modifying code
> since at least module loaders became a thing -- that's almost 40 years.
> Today, especially for risc-v, Linux is aggressively self-modifying; there's
> no option for some risc-v SoC if you want them to work correctly. Linux
> rewrites the entire kernel text in early boot stages. You can consider the
> last stage linker optimization occurs in Linux early boot code.
>
> On Thu, Sep 19, 2024 at 12:13 AM Warner Losh <imp@bsdimp.com> wrote:
>
>>
>>
>> On Thu, Sep 19, 2024, 1:05 AM Bakul Shah via TUHS <tuhs@tuhs.org> wrote:
>>
>>> Can you not avoid resetting the machine? This can be treated almost as
>>> sleep in the old kernel, wakeup in the new one! You do have to reset
>>> devices individually (which may not always work if it requires assistance
>>> from some undocumented firmware).
>>>
>>
>> Kexec does just this. The new kernel boots without going through the
>> reset vector. The old kernel keeps a tiny bit of code around that tears
>> down all the protections, etc and hands off to the new kernel a mostly
>> reset machine.. but it doesn't go through the firmware to do it... it was
>> the original reason for it in linux: fast reboot times.
>>
>> Warner
>>
>> On Sep 18, 2024, at 4:58 PM, ron minnich <rminnich@gmail.com> wrote:
>>>
>>> well, yes, on many systems, there's a lot that runs before the kernel.
>>> But if you have a risc-v system with oreboot, you own the system. The
>>> problem is that on most of these systems a reset will stop the dram clock
>>> for a little bit, or glitch clock enable, or dram power, or whatever. New
>>> systems are not designed to allow this.
>>>
>>> Ideally, we could force a reset of everything save memory, but modern
>>> systems are not designed in this way. Most annoying.
>>>
>>> On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org> wrote:
>>>
>>>> I would prefer old kernel to new kernel handoff if it can be made to
>>>> work reliably. Nowadays there are a lot of things that run before the
>>>> kernel gets control.
>>>>
>>>> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
>>>>
>>>> Interesting about the amiga. I'm assuming their firmware zeros memory
>>>> on reset, so you have to do handoff from kernel to kernel, not via a reset
>>>> and so on?
>>>>
>>>> What was particularly nice about the V6/PDP-11 case: we were able to
>>>> yank reset, which let us cleanly reset/disable devices, because everything
>>>> was in memory when we got back. I miss the simplicity of the old machines.
>>>>
>>>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org>
>>>> wrote:
>>>>
>>>>>
>>>>> We had/have this functionality in the Amiga port of NetBSD.
>>>>>
>>>>> It is implemented as `/dev/reload` device and you copy a kernel image
>>>>> to it. In locore.s there's code that copies the kernel image over top of
>>>>> the running kernel and then restarts. I believe for it to work nothing
>>>>> below the copy code in locore.s can change :)
>>>>>
>>>>> Thanks,
>>>>> Chris.
>>>>>
>>>>> Phil Budne <phil@ultimate.com> writes:
>>>>>
>>>>> > ron minnich wrote:
>>>>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots
>>>>> kernel" or
>>>>> >> was there something before?
>>>>> >
>>>>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>>>>> > that reboots the system by reading a.out into user memory (in the
>>>>> high
>>>>> > 4K of core), then copies it to low memory and jumping to the entry
>>>>> > point.  The name suggests its original intended use was to test a new
>>>>> > system (kernel).
>>>>> >
>>>>> > P.S.
>>>>> > Normal bootable system images seem to have been stored in reserved
>>>>> > tracks of the (fixed head) disk (that are inacessible via system
>>>>> calls):
>>>>> >
>>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>>>>> > reads a.out and uses I/O instructions to write it out.
>>>>> >
>>>>> > P.P.S.
>>>>> > Accordingly, I put together a "paper tape" for booting the system:
>>>>> >
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>>>>> >
>>>>> > P.P.P.S.
>>>>> > The system (kernel) is 3K words, the last 1K of low memory
>>>>> > used for the character table for the vector graphics controller.
>>>>> >
>>>>> > The definitions for the table are compiled by
>>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>>>>> > from definition file
>>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>>>>> > (after, ISTR, figuring out the ordering of the listing pages!)
>>>>> >
>>>>> > I don't think we ever figured out how the initial character table
>>>>> > is loaded into core.  One thing that was missing from the table
>>>>> > was the dispatch array, which I recreated:
>>>>> >
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>>>>> >
>>>>> > The system (kernel) could be built for a "cold start", reloading the
>>>>> > disk (prone to head crashes?) from paper tape? But I don't think
>>>>> > anyone ever reconstructed the procedure for rebuilding a disk that
>>>>> way.
>>>>> >
>>>>> > The disk was two sided, and the running system only used one side:
>>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>>>>> > appear to be programs to save and restore the filesystem from the
>>>>> > "other" side of the disk.
>>>>>
>>>>>
>>>>
>>>

[-- Attachment #2: Type: text/html, Size: 13028 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Alan Coopersmith via TUHS]

On 9/19/24 00:13, Warner Losh wrote:
> Kexec does just this. The new kernel boots without going through the reset 
> vector. The old kernel keeps a tiny bit of code around that tears down all the 
> protections, etc and hands off to the new kernel a mostly reset machine.. but it 
> doesn't go through the firmware to do it... it was the original reason for it in 
> linux: fast reboot times.

Indeed - when Sun implemented the equivalent in Solaris, the feature was
literally named "Fast Reboot":

https://illumos.org/opensolaris/ARChive/PSARC/2008/382/final.materials/fastboot.txt

It will refuse to run and require a trip through the firmware if any active
drivers don't support or can't complete a "quiesce" routine to finish off
any in-progress operations and reset the the hardware to the state expected
for a fresh boot:

https://illumos.org/opensolaris/ARChive/PSARC/2008/382/final.materials/quiesce.man.txt

-- 
         -Alan Coopersmith-                 alan.coopersmith@oracle.com
          Oracle Solaris Engineering - https://blogs.oracle.com/solaris

[TUHS] Re: kernel boots kernel in 1977

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 8187 bytes --]

to reiterate: you can avoid resetting the machine, and for all the x
million systems in data centers around the world, we do avoid resetting the
machine.

But that comes with its own set of issues, including kernel version x not
being able to boot kernel version y (very common with linux, no problem on
plan 9); and hardware not behaving well, since few people write drivers
that properly reset hardware; or the hardware can't be cleaned up absent a
reset (most common problem areas are NICs and graphics). Very few linux
drivers can properly shut down hardware for a kexec. The IOMMU and MSIx
added a whole new world of fun. Sometimes it feels like it works by
accident.

Also recall that side channels across a kexec are an issue that has to be
considered. At Google we've considered them by, e.g., turning on "zero on
free" and "zero on alloc" in the kernel that will kexec, only using a small
amount of memory in the first kernel (32GiB is small! Ha!), among other
things. But since DRAM SPD and Voltage regulator module FLASH provide
places to hide things, it's getting messy.

So it's not as simple as "reset bad, not reset good". Hardware is poorly
designed, or the drivers are poorly written, and absent a reset, the
kexec'ed kernel may fail to boot -- lockup is common, panic is common. That
said, no system I know of implements kexec with a reset in the middle.

Short history: in the Linux world, kernel boots kernel was done, in 1999,
by LOBOS (me) and Eric Hendriks (Two Kernel Monte), and Alpha Power
(DBLX). Werner Almesberger did his own thing ca. 2000 called (iirc)
bootimg. Eric Biederman looked at LOBOS, did not like it, and wrote kexec,
I believe around 2001. Plan 9 got kernel boots kernel around that time. As
usual, the Plan 9 implementation was the most compact and cleanest. This
paper https://ieeexplore.ieee.org/document/1392643 compares them.

The AlphaPower DBLX code was lost when the company went under. They made a
heroic effort to get it to sourceforge but things happened too fast. DBLX
means "direct boot linux" -- the acronym reads better.

The first kexec was a very general interface, with a Himalayan learning
curve. At some point an Intel engineer found kexec confusing and wrote an
entirely new type of kexec, with a different API, that many people found
easier.

so kexec has been around for 20 years, and we're still getting the hang of
it, and there are still people who claim that it will never fully work.

Anyway, we're far afield of the original question, but it was very
interesting to read how far back the idea goes! PDP-7, who knew?

p.s. as to an unrelated discussion: kernels have been self modifying code
since at least module loaders became a thing -- that's almost 40 years.
Today, especially for risc-v, Linux is aggressively self-modifying; there's
no option for some risc-v SoC if you want them to work correctly. Linux
rewrites the entire kernel text in early boot stages. You can consider the
last stage linker optimization occurs in Linux early boot code.

On Thu, Sep 19, 2024 at 12:13 AM Warner Losh <imp@bsdimp.com> wrote:

>
>
> On Thu, Sep 19, 2024, 1:05 AM Bakul Shah via TUHS <tuhs@tuhs.org> wrote:
>
>> Can you not avoid resetting the machine? This can be treated almost as
>> sleep in the old kernel, wakeup in the new one! You do have to reset
>> devices individually (which may not always work if it requires assistance
>> from some undocumented firmware).
>>
>
> Kexec does just this. The new kernel boots without going through the reset
> vector. The old kernel keeps a tiny bit of code around that tears down all
> the protections, etc and hands off to the new kernel a mostly reset
> machine.. but it doesn't go through the firmware to do it... it was the
> original reason for it in linux: fast reboot times.
>
> Warner
>
> On Sep 18, 2024, at 4:58 PM, ron minnich <rminnich@gmail.com> wrote:
>>
>> well, yes, on many systems, there's a lot that runs before the kernel.
>> But if you have a risc-v system with oreboot, you own the system. The
>> problem is that on most of these systems a reset will stop the dram clock
>> for a little bit, or glitch clock enable, or dram power, or whatever. New
>> systems are not designed to allow this.
>>
>> Ideally, we could force a reset of everything save memory, but modern
>> systems are not designed in this way. Most annoying.
>>
>> On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org> wrote:
>>
>>> I would prefer old kernel to new kernel handoff if it can be made to
>>> work reliably. Nowadays there are a lot of things that run before the
>>> kernel gets control.
>>>
>>> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
>>>
>>> Interesting about the amiga. I'm assuming their firmware zeros memory on
>>> reset, so you have to do handoff from kernel to kernel, not via a reset and
>>> so on?
>>>
>>> What was particularly nice about the V6/PDP-11 case: we were able to
>>> yank reset, which let us cleanly reset/disable devices, because everything
>>> was in memory when we got back. I miss the simplicity of the old machines.
>>>
>>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org>
>>> wrote:
>>>
>>>>
>>>> We had/have this functionality in the Amiga port of NetBSD.
>>>>
>>>> It is implemented as `/dev/reload` device and you copy a kernel image
>>>> to it. In locore.s there's code that copies the kernel image over top of
>>>> the running kernel and then restarts. I believe for it to work nothing
>>>> below the copy code in locore.s can change :)
>>>>
>>>> Thanks,
>>>> Chris.
>>>>
>>>> Phil Budne <phil@ultimate.com> writes:
>>>>
>>>> > ron minnich wrote:
>>>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots
>>>> kernel" or
>>>> >> was there something before?
>>>> >
>>>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>>>> > that reboots the system by reading a.out into user memory (in the high
>>>> > 4K of core), then copies it to low memory and jumping to the entry
>>>> > point.  The name suggests its original intended use was to test a new
>>>> > system (kernel).
>>>> >
>>>> > P.S.
>>>> > Normal bootable system images seem to have been stored in reserved
>>>> > tracks of the (fixed head) disk (that are inacessible via system
>>>> calls):
>>>> >
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>>>> > reads a.out and uses I/O instructions to write it out.
>>>> >
>>>> > P.P.S.
>>>> > Accordingly, I put together a "paper tape" for booting the system:
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>>>> >
>>>> > P.P.P.S.
>>>> > The system (kernel) is 3K words, the last 1K of low memory
>>>> > used for the character table for the vector graphics controller.
>>>> >
>>>> > The definitions for the table are compiled by
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>>>> > from definition file
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>>>> > (after, ISTR, figuring out the ordering of the listing pages!)
>>>> >
>>>> > I don't think we ever figured out how the initial character table
>>>> > is loaded into core.  One thing that was missing from the table
>>>> > was the dispatch array, which I recreated:
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>>>> >
>>>> > The system (kernel) could be built for a "cold start", reloading the
>>>> > disk (prone to head crashes?) from paper tape? But I don't think
>>>> > anyone ever reconstructed the procedure for rebuilding a disk that
>>>> way.
>>>> >
>>>> > The disk was two sided, and the running system only used one side:
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>>>> > appear to be programs to save and restore the filesystem from the
>>>> > "other" side of the disk.
>>>>
>>>>
>>>
>>

[-- Attachment #2: Type: text/html, Size: 11572 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 4905 bytes --]

On Thu, Sep 19, 2024, 1:05 AM Bakul Shah via TUHS <tuhs@tuhs.org> wrote:

> Can you not avoid resetting the machine? This can be treated almost as
> sleep in the old kernel, wakeup in the new one! You do have to reset
> devices individually (which may not always work if it requires assistance
> from some undocumented firmware).
>

Kexec does just this. The new kernel boots without going through the reset
vector. The old kernel keeps a tiny bit of code around that tears down all
the protections, etc and hands off to the new kernel a mostly reset
machine.. but it doesn't go through the firmware to do it... it was the
original reason for it in linux: fast reboot times.

Warner

On Sep 18, 2024, at 4:58 PM, ron minnich <rminnich@gmail.com> wrote:
>
> well, yes, on many systems, there's a lot that runs before the kernel. But
> if you have a risc-v system with oreboot, you own the system. The problem
> is that on most of these systems a reset will stop the dram clock for a
> little bit, or glitch clock enable, or dram power, or whatever. New systems
> are not designed to allow this.
>
> Ideally, we could force a reset of everything save memory, but modern
> systems are not designed in this way. Most annoying.
>
> On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org> wrote:
>
>> I would prefer old kernel to new kernel handoff if it can be made to work
>> reliably. Nowadays there are a lot of things that run before the kernel
>> gets control.
>>
>> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
>>
>> Interesting about the amiga. I'm assuming their firmware zeros memory on
>> reset, so you have to do handoff from kernel to kernel, not via a reset and
>> so on?
>>
>> What was particularly nice about the V6/PDP-11 case: we were able to yank
>> reset, which let us cleanly reset/disable devices, because everything was
>> in memory when we got back. I miss the simplicity of the old machines.
>>
>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org>
>> wrote:
>>
>>>
>>> We had/have this functionality in the Amiga port of NetBSD.
>>>
>>> It is implemented as `/dev/reload` device and you copy a kernel image to
>>> it. In locore.s there's code that copies the kernel image over top of the
>>> running kernel and then restarts. I believe for it to work nothing below
>>> the copy code in locore.s can change :)
>>>
>>> Thanks,
>>> Chris.
>>>
>>> Phil Budne <phil@ultimate.com> writes:
>>>
>>> > ron minnich wrote:
>>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots
>>> kernel" or
>>> >> was there something before?
>>> >
>>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>>> > that reboots the system by reading a.out into user memory (in the high
>>> > 4K of core), then copies it to low memory and jumping to the entry
>>> > point.  The name suggests its original intended use was to test a new
>>> > system (kernel).
>>> >
>>> > P.S.
>>> > Normal bootable system images seem to have been stored in reserved
>>> > tracks of the (fixed head) disk (that are inacessible via system
>>> calls):
>>> >
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>>> > reads a.out and uses I/O instructions to write it out.
>>> >
>>> > P.P.S.
>>> > Accordingly, I put together a "paper tape" for booting the system:
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>>> >
>>> > P.P.P.S.
>>> > The system (kernel) is 3K words, the last 1K of low memory
>>> > used for the character table for the vector graphics controller.
>>> >
>>> > The definitions for the table are compiled by
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>>> > from definition file
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>>> > (after, ISTR, figuring out the ordering of the listing pages!)
>>> >
>>> > I don't think we ever figured out how the initial character table
>>> > is loaded into core.  One thing that was missing from the table
>>> > was the dispatch array, which I recreated:
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>>> >
>>> > The system (kernel) could be built for a "cold start", reloading the
>>> > disk (prone to head crashes?) from paper tape? But I don't think
>>> > anyone ever reconstructed the procedure for rebuilding a disk that way.
>>> >
>>> > The disk was two sided, and the running system only used one side:
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>>> > appear to be programs to save and restore the filesystem from the
>>> > "other" side of the disk.
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 7832 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Bakul Shah via TUHS]

> On Sep 18, 2024, at 5:53 PM, Dan Cross <crossd@gmail.com> wrote:
> 
> On Wed, Sep 18, 2024 at 8:05 PM Bakul Shah via TUHS <tuhs@tuhs.org> wrote:
>> Can you not avoid resetting the machine? This can be treated almost as sleep in the old kernel, wakeup in the new one! You do have to reset devices individually (which may not always work if it requires assistance from some undocumented firmware).
> 
> Perhaps this is what you mean when you mention assistance from
> firmware, Bakul, but it may be useful to consider that _many_ devices
> are touched by e.g. a BIOS or UEFI or whatever well before the OS is
> even loaded.

Right but presumably the old kernel leaves them in a good enough state.

> If one steps back and considers the utility of a BIOS/UEFI (and I
> often lump these into the same category), there are three principal
> reasons for it: 1) back in the bad old days, we could offload common
> IO functions into code stored on a ROM, freeing up precious RAM for
> programs. 2) firmware provides a layer of indirection between the
> system and the host software, allowing both to vary while continuing
> to work with newer versions of the other. And finally 3) firmware
> facilitates bootstrapping the system by providing the host some way to
> access devices and locate and load an OS image, er, before the OS
> image is loaded. SOMETHING has to get enough code loaded from
> somewhere to start the system; often times that's firmware.

The new OS image is already in memory but may need to be copied to
the right place. The devices were already working (but may need to
have their interrupts disabled and any DMA stopped etc.).

> Anyway, the last two suggest that device state can be arbitrarily
> munged before the OS takes over, and an actual reset at the device
> level might wipe out some state the OS depends on. Consider, for
> example, programming PCI BARs; on a "modern" x86-64 system with UEFI,
> this is done by firmware in the PEI layer, and the OS may expect that
> to already be set up by the time it is probing buses. An actual
> honest-to-goodness reset will probably wipe the BARs, requiring the
> host OS to program them (ironically, many OSes are already equipped to
> do so, as they have to handle these cases for e.g. PCI hotplug events,
> though many don't do it in the "ordinary" discovery and initialization
> phase of boot).

All that is done on powerup.

> 
> I suppose the point is that a reset is great because it really does
> wipe out state, but it may also be a bummer because, well, it really
> does wipe out state. :-)

:-) I was speculating that kernel to kernel warmboot should be doable.


> 
>        - Dan C.
> 
>> On Sep 18, 2024, at 4:58 PM, ron minnich <rminnich@gmail.com> wrote:
>> 
>> well, yes, on many systems, there's a lot that runs before the kernel. But if you have a risc-v system with oreboot, you own the system. The problem is that on most of these systems a reset will stop the dram clock for a little bit, or glitch clock enable, or dram power, or whatever. New systems are not designed to allow this.
>> 
>> Ideally, we could force a reset of everything save memory, but modern systems are not designed in this way. Most annoying.
>> 
>> On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org> wrote:
>>> 
>>> I would prefer old kernel to new kernel handoff if it can be made to work reliably. Nowadays there are a lot of things that run before the kernel gets control.
>>> 
>>> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
>>> 
>>> Interesting about the amiga. I'm assuming their firmware zeros memory on reset, so you have to do handoff from kernel to kernel, not via a reset and so on?
>>> 
>>> What was particularly nice about the V6/PDP-11 case: we were able to yank reset, which let us cleanly reset/disable devices, because everything was in memory when we got back. I miss the simplicity of the old machines.
>>> 
>>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org> wrote:
>>>> 
>>>> 
>>>> We had/have this functionality in the Amiga port of NetBSD.
>>>> 
>>>> It is implemented as `/dev/reload` device and you copy a kernel image to it. In locore.s there's code that copies the kernel image over top of the running kernel and then restarts. I believe for it to work nothing below the copy code in locore.s can change :)
>>>> 
>>>> Thanks,
>>>> Chris.
>>>> 
>>>> Phil Budne <phil@ultimate.com> writes:
>>>> 
>>>>> ron minnich wrote:
>>>>>> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
>>>>>> was there something before?
>>>>> 
>>>>> There was!  The PDP-7 UNIX listings contain a program trysys.s
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>>>>> that reboots the system by reading a.out into user memory (in the high
>>>>> 4K of core), then copies it to low memory and jumping to the entry
>>>>> point.  The name suggests its original intended use was to test a new
>>>>> system (kernel).
>>>>> 
>>>>> P.S.
>>>>> Normal bootable system images seem to have been stored in reserved
>>>>> tracks of the (fixed head) disk (that are inacessible via system calls):
>>>>> 
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>>>>> reads a.out and uses I/O instructions to write it out.
>>>>> 
>>>>> P.P.S.
>>>>> Accordingly, I put together a "paper tape" for booting the system:
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>>>>> 
>>>>> P.P.P.S.
>>>>> The system (kernel) is 3K words, the last 1K of low memory
>>>>> used for the character table for the vector graphics controller.
>>>>> 
>>>>> The definitions for the table are compiled by
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>>>>> from definition file
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>>>>> (after, ISTR, figuring out the ordering of the listing pages!)
>>>>> 
>>>>> I don't think we ever figured out how the initial character table
>>>>> is loaded into core.  One thing that was missing from the table
>>>>> was the dispatch array, which I recreated:
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>>>>> 
>>>>> The system (kernel) could be built for a "cold start", reloading the
>>>>> disk (prone to head crashes?) from paper tape? But I don't think
>>>>> anyone ever reconstructed the procedure for rebuilding a disk that way.
>>>>> 
>>>>> The disk was two sided, and the running system only used one side:
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>>>>> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>>>>> appear to be programs to save and restore the filesystem from the
>>>>> "other" side of the disk.
>>>> 
>>> 
>> 

[TUHS] Re: kernel boots kernel in 1977

[Dan Cross]

On Wed, Sep 18, 2024 at 8:32 PM George Michaelson <ggm@algebras.org> wrote:
> I think I misunderstand a lot of this Dan, so I won't try to prosecute
> my case. I felt your answer went to the ring model separating user
> space from kernel space, but not the ring model embedded in the CPU,
> down in the chip register set and instructions.

Perhaps I don't understand what you mean, then. How are those things
different? Or, put another way, one requires the other: the thing that
separates userspace and kernel space _is_ the CPU's model of
protection domains, but these have different names on different CPUs.
Sure, they're numbered rings on x86 (0 being "kernel" mode, and 3
"user", with the rarely used 1 and 2 all but ignored outside of, I
think, OS/2 2.x), but the analogous concept is called "access modes"
of execution on VAX; RISC-V and ARM retain the "mode" nomenclature, as
do most RISC architectures (MIPS, Alpha, SPARC, etc), but "kernel
mode" (ring 0) is variously called "supervisor", "kernel", "executive"
or "privileged" mode in different places (the VAX, of course, had 4
modes: kernel, executive, supervisor, and user; and the GE 645 had 9).

> "trusted" computing was the idea across boot you get to initialise
> some state and then set a flag, a bit, which forces the kernel to live
> within the constraint set encoded from that transition.

Do you mean something like ARM's trustzone? That's different, and
usually involves a "trusted executive" outside of the control of the
OS. But that doesn't a priori mean that the OS can't do funky things
like replace itself.

> Loading a new
> kernel imples writing into things which I believe(d) you had been
> constrained not to do.
>
> The chip level rings are meant to be absolutes.

Not really. What would that mean? How would loadable kernel modules
work in such a scenario? What about self-modifying code (when Linux
boots up, it will examine what machine it's running on and quite
possibly binary patch some hot execution paths to avoid branches)? If
one can't move between processor access modes in some controlled way,
how would IO or system calls work?

As for trust-zone-y type stuff, consider that part of the overall
security posture of a system may be to measure whatever is presented
to the kexec mechanism and verify that it's properly signed; if it is,
it's ok to boot it; if not, the operation fails. If the intent is
merely to prevent untrusted code from running in kernel mode, such a
mechanism (with a lot of details omitted) will more or less or that,
provided the lowest-level trusted bootstrap code made sure that the
first OS loaded was properly signed. This seems like a reasonable
design that shouldn't be prohibited.

> Warm boot vs Cold Boot vs .. I dunno, a manual transition through GPT
> and BIOS states to change things?

This is something people forget: the BIOS is just code. There's
nothing magical about it. If you're talking about something like
Trustzone to ensure that the _BIOS_ is properly trusted, then that's a
separate matter, but mostly orthogonal to the kexec thing.

> I don't do this for a living. I stress I very probably completely
> mis-understand what is a pledge to good faith and what is actually
> meant to be enforced by hardware.

Hey, no problem: the goal of this list is to explore the history and
(one hopes) learn from it as well. These are subtle topics and I'm
sure we all, myself very much included, have much to learn here. I
think we're all approaching this with the same collegial spirit of
inquiry!

        - Dan C.

[TUHS] Re: kernel boots kernel in 1977

[Bakul Shah via TUHS]

Forget "rings". Unix needs only two states from hardware.
Forget virtualization for now as well.

The kernel runs in the supervisor state, the user code in the
unprivileged state where it can't execute or see certain
instructions or processor state or peripherals and must make
system calls for controlled access to the same.

In addition Unix has the "root" user that can access much more
but it too must go through the system call interface.

A "warm reboot" system call would simply arrange things so that
the new kernel image is copied in the right place and control
will eventually pass to it.

It is not so simple these days as the hardware is much more
complex and often requires vendor provided firmware assist
to properly initialize the system before control gets passed
to an OS kernel but no change in the protection model is
required for a kernel to kernel handoff.

> On Sep 18, 2024, at 5:04 PM, George Michaelson <ggm@algebras.org> wrote:
> 
> Not to be "that guy" some of this can be read as "the whole
> ring/protection model is a myth" because doing a boot into a new
> executive demands "writing" state into parts of the system which
> people believe by a phenomenal act of faith are "protected" against
> that.
> 
> Virtualisation makes much of this latent "protection rings are a bit
> of a myth" concrete.
> 
> Maybe I misunderstand some of this. I can believe that UNIX-like
> things try to work irrespective of what chip designers do underneath
> to construct things like TPM, and talk to it in the limited ways
> necessary.
> 
> -G

[TUHS] Re: kernel boots kernel in 1977

[Dan Cross]

On Wed, Sep 18, 2024 at 8:05 PM Bakul Shah via TUHS <tuhs@tuhs.org> wrote:
> Can you not avoid resetting the machine? This can be treated almost as sleep in the old kernel, wakeup in the new one! You do have to reset devices individually (which may not always work if it requires assistance from some undocumented firmware).

Perhaps this is what you mean when you mention assistance from
firmware, Bakul, but it may be useful to consider that _many_ devices
are touched by e.g. a BIOS or UEFI or whatever well before the OS is
even loaded.

If one steps back and considers the utility of a BIOS/UEFI (and I
often lump these into the same category), there are three principal
reasons for it: 1) back in the bad old days, we could offload common
IO functions into code stored on a ROM, freeing up precious RAM for
programs. 2) firmware provides a layer of indirection between the
system and the host software, allowing both to vary while continuing
to work with newer versions of the other. And finally 3) firmware
facilitates bootstrapping the system by providing the host some way to
access devices and locate and load an OS image, er, before the OS
image is loaded. SOMETHING has to get enough code loaded from
somewhere to start the system; often times that's firmware.

Anyway, the last two suggest that device state can be arbitrarily
munged before the OS takes over, and an actual reset at the device
level might wipe out some state the OS depends on. Consider, for
example, programming PCI BARs; on a "modern" x86-64 system with UEFI,
this is done by firmware in the PEI layer, and the OS may expect that
to already be set up by the time it is probing buses. An actual
honest-to-goodness reset will probably wipe the BARs, requiring the
host OS to program them (ironically, many OSes are already equipped to
do so, as they have to handle these cases for e.g. PCI hotplug events,
though many don't do it in the "ordinary" discovery and initialization
phase of boot).

I suppose the point is that a reset is great because it really does
wipe out state, but it may also be a bummer because, well, it really
does wipe out state. :-)

        - Dan C.

> On Sep 18, 2024, at 4:58 PM, ron minnich <rminnich@gmail.com> wrote:
>
> well, yes, on many systems, there's a lot that runs before the kernel. But if you have a risc-v system with oreboot, you own the system. The problem is that on most of these systems a reset will stop the dram clock for a little bit, or glitch clock enable, or dram power, or whatever. New systems are not designed to allow this.
>
> Ideally, we could force a reset of everything save memory, but modern systems are not designed in this way. Most annoying.
>
> On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org> wrote:
>>
>> I would prefer old kernel to new kernel handoff if it can be made to work reliably. Nowadays there are a lot of things that run before the kernel gets control.
>>
>> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
>>
>> Interesting about the amiga. I'm assuming their firmware zeros memory on reset, so you have to do handoff from kernel to kernel, not via a reset and so on?
>>
>> What was particularly nice about the V6/PDP-11 case: we were able to yank reset, which let us cleanly reset/disable devices, because everything was in memory when we got back. I miss the simplicity of the old machines.
>>
>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org> wrote:
>>>
>>>
>>> We had/have this functionality in the Amiga port of NetBSD.
>>>
>>> It is implemented as `/dev/reload` device and you copy a kernel image to it. In locore.s there's code that copies the kernel image over top of the running kernel and then restarts. I believe for it to work nothing below the copy code in locore.s can change :)
>>>
>>> Thanks,
>>> Chris.
>>>
>>> Phil Budne <phil@ultimate.com> writes:
>>>
>>> > ron minnich wrote:
>>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
>>> >> was there something before?
>>> >
>>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>>> > that reboots the system by reading a.out into user memory (in the high
>>> > 4K of core), then copies it to low memory and jumping to the entry
>>> > point.  The name suggests its original intended use was to test a new
>>> > system (kernel).
>>> >
>>> > P.S.
>>> > Normal bootable system images seem to have been stored in reserved
>>> > tracks of the (fixed head) disk (that are inacessible via system calls):
>>> >
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>>> > reads a.out and uses I/O instructions to write it out.
>>> >
>>> > P.P.S.
>>> > Accordingly, I put together a "paper tape" for booting the system:
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>>> >
>>> > P.P.P.S.
>>> > The system (kernel) is 3K words, the last 1K of low memory
>>> > used for the character table for the vector graphics controller.
>>> >
>>> > The definitions for the table are compiled by
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>>> > from definition file
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>>> > (after, ISTR, figuring out the ordering of the listing pages!)
>>> >
>>> > I don't think we ever figured out how the initial character table
>>> > is loaded into core.  One thing that was missing from the table
>>> > was the dispatch array, which I recreated:
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>>> >
>>> > The system (kernel) could be built for a "cold start", reloading the
>>> > disk (prone to head crashes?) from paper tape? But I don't think
>>> > anyone ever reconstructed the procedure for rebuilding a disk that way.
>>> >
>>> > The disk was two sided, and the running system only used one side:
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>>> > appear to be programs to save and restore the filesystem from the
>>> > "other" side of the disk.
>>>
>>
>

[TUHS] Re: kernel boots kernel in 1977

[George Michaelson]

I think I misunderstand a lot of this Dan, so I won't try to prosecute
my case. I felt your answer went to the ring model separating user
space from kernel space, but not the ring model embedded in the CPU,
down in the chip register set and instructions.

"trusted" computing was the idea across boot you get to initialise
some state and then set a flag, a bit, which forces the kernel to live
within the constraint set encoded from that transition. Loading a new
kernel imples writing into things which I believe(d) you had been
constrained not to do.

The chip level rings are meant to be absolutes.

Warm boot vs Cold Boot vs .. I dunno, a manual transition through GPT
and BIOS states to change things?

I don't do this for a living. I stress I very probably completely
mis-understand what is a pledge to good faith and what is actually
meant to be enforced by hardware.

-G

[TUHS] Re: kernel boots kernel in 1977

[Dan Cross]

On Wed, Sep 18, 2024 at 8:11 PM George Michaelson <ggm@algebras.org> wrote:
> Not to be "that guy" some of this can be read as "the whole
> ring/protection model is a myth" because doing a boot into a new
> executive demands "writing" state into parts of the system which
> people believe by a phenomenal act of faith are "protected" against
> that.

I'm not sure that I completely buy that. The act of replacing the
kernel is, of course, a privileged operation. In some abstract sense,
this is little different than triggering a reboot via a system call:
in both cases, the system as we know it goes away and is replaced by
something else.

Similarly, even in the cold-boot case, very early on in the machine's
lifecycle the kernel image is found and loaded into memory for
execution. What's the difference if it's a bootloader that does that,
or the kernel itself? Indeed, from the perspective of the newly booted
kernel, the prior execution of the machine can be regarded as a
single, perhaps exceptionally long, boot loading phase.

> Virtualisation makes much of this latent "protection rings are a bit
> of a myth" concrete.

How so? Virtualization doesn't all of a sudden mean that (guest)
userspace can scribble all over (guest) kernel memory, let alone the
host. The deluge of recent speculation bugs notwithstanding, the
separation between user and executive is still very much intrinsic to
the proper operation of modern computers.

> Maybe I misunderstand some of this. I can believe that UNIX-like
> things try to work irrespective of what chip designers do underneath
> to construct things like TPM, and talk to it in the limited ways
> necessary.

Yeah, we have seams between the host OS and the actual computer
mediated by firmware, but given the state of that firmware, the
present situation is borderline untenable. That's not to say that
interfaces are bad, or that they're not useful, but the interfaces
that we presently have are not good. E.g.,
https://www.usenix.org/conference/osdi21/presentation/fri-keynote

At work, we've taken this to heart; the first x86 instruction when our
CPUs come out of reset is actually sitting in code we wrote, and we
boot directly into the operating system. https://vimeo.com/756050840,
https://github.com/oxidecomputer/phbl, and
https://rfd.shared.oxide.computer/rfd/0284 are interesting references.

        - Dan C.

[TUHS] Re: kernel boots kernel in 1977

[Bakul Shah via TUHS]

[-- Attachment #1: Type: text/plain, Size: 4565 bytes --]

Can you not avoid resetting the machine? This can be treated almost as sleep in the old kernel, wakeup in the new one! You do have to reset devices individually (which may not always work if it requires assistance from some undocumented firmware).

> On Sep 18, 2024, at 4:58 PM, ron minnich <rminnich@gmail.com> wrote:
> 
> well, yes, on many systems, there's a lot that runs before the kernel. But if you have a risc-v system with oreboot, you own the system. The problem is that on most of these systems a reset will stop the dram clock for a little bit, or glitch clock enable, or dram power, or whatever. New systems are not designed to allow this.
> 
> Ideally, we could force a reset of everything save memory, but modern systems are not designed in this way. Most annoying.
> 
> On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org <mailto:bakul@iitbombay.org>> wrote:
>> I would prefer old kernel to new kernel handoff if it can be made to work reliably. Nowadays there are a lot of things that run before the kernel gets control. 
>> 
>>> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com <mailto:rminnich@gmail.com>> wrote:
>>> 
>>> Interesting about the amiga. I'm assuming their firmware zeros memory on reset, so you have to do handoff from kernel to kernel, not via a reset and so on?
>>> 
>>> What was particularly nice about the V6/PDP-11 case: we were able to yank reset, which let us cleanly reset/disable devices, because everything was in memory when we got back. I miss the simplicity of the old machines.
>>> 
>>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org <mailto:chopps@chopps.org>> wrote:
>>>> 
>>>> We had/have this functionality in the Amiga port of NetBSD.
>>>> 
>>>> It is implemented as `/dev/reload` device and you copy a kernel image to it. In locore.s there's code that copies the kernel image over top of the running kernel and then restarts. I believe for it to work nothing below the copy code in locore.s can change :)
>>>> 
>>>> Thanks,
>>>> Chris.
>>>> 
>>>> Phil Budne <phil@ultimate.com <mailto:phil@ultimate.com>> writes:
>>>> 
>>>> > ron minnich wrote:
>>>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
>>>> >> was there something before?
>>>> >
>>>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>>>> > that reboots the system by reading a.out into user memory (in the high
>>>> > 4K of core), then copies it to low memory and jumping to the entry
>>>> > point.  The name suggests its original intended use was to test a new
>>>> > system (kernel).
>>>> >
>>>> > P.S.
>>>> > Normal bootable system images seem to have been stored in reserved
>>>> > tracks of the (fixed head) disk (that are inacessible via system calls):
>>>> >
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>>>> > reads a.out and uses I/O instructions to write it out.
>>>> >
>>>> > P.P.S.
>>>> > Accordingly, I put together a "paper tape" for booting the system:
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>>>> >
>>>> > P.P.P.S.
>>>> > The system (kernel) is 3K words, the last 1K of low memory
>>>> > used for the character table for the vector graphics controller.
>>>> >
>>>> > The definitions for the table are compiled by
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>>>> > from definition file
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>>>> > (after, ISTR, figuring out the ordering of the listing pages!)
>>>> >
>>>> > I don't think we ever figured out how the initial character table
>>>> > is loaded into core.  One thing that was missing from the table
>>>> > was the dispatch array, which I recreated:
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>>>> >
>>>> > The system (kernel) could be built for a "cold start", reloading the
>>>> > disk (prone to head crashes?) from paper tape? But I don't think
>>>> > anyone ever reconstructed the procedure for rebuilding a disk that way.
>>>> >
>>>> > The disk was two sided, and the running system only used one side:
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>>>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>>>> > appear to be programs to save and restore the filesystem from the
>>>> > "other" side of the disk.
>>>> 
>> 


[-- Attachment #2: Type: text/html, Size: 6723 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[George Michaelson]

Not to be "that guy" some of this can be read as "the whole
ring/protection model is a myth" because doing a boot into a new
executive demands "writing" state into parts of the system which
people believe by a phenomenal act of faith are "protected" against
that.

Virtualisation makes much of this latent "protection rings are a bit
of a myth" concrete.

Maybe I misunderstand some of this. I can believe that UNIX-like
things try to work irrespective of what chip designers do underneath
to construct things like TPM, and talk to it in the limited ways
necessary.

-G

[TUHS] Re: kernel boots kernel in 1977

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 4018 bytes --]

well, yes, on many systems, there's a lot that runs before the kernel. But
if you have a risc-v system with oreboot, you own the system. The problem
is that on most of these systems a reset will stop the dram clock for a
little bit, or glitch clock enable, or dram power, or whatever. New systems
are not designed to allow this.

Ideally, we could force a reset of everything save memory, but modern
systems are not designed in this way. Most annoying.

On Wed, Sep 18, 2024 at 4:38 PM Bakul Shah <bakul@iitbombay.org> wrote:

> I would prefer old kernel to new kernel handoff if it can be made to work
> reliably. Nowadays there are a lot of things that run before the kernel
> gets control.
>
> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
>
> Interesting about the amiga. I'm assuming their firmware zeros memory on
> reset, so you have to do handoff from kernel to kernel, not via a reset and
> so on?
>
> What was particularly nice about the V6/PDP-11 case: we were able to yank
> reset, which let us cleanly reset/disable devices, because everything was
> in memory when we got back. I miss the simplicity of the old machines.
>
> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org> wrote:
>
>>
>> We had/have this functionality in the Amiga port of NetBSD.
>>
>> It is implemented as `/dev/reload` device and you copy a kernel image to
>> it. In locore.s there's code that copies the kernel image over top of the
>> running kernel and then restarts. I believe for it to work nothing below
>> the copy code in locore.s can change :)
>>
>> Thanks,
>> Chris.
>>
>> Phil Budne <phil@ultimate.com> writes:
>>
>> > ron minnich wrote:
>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots
>> kernel" or
>> >> was there something before?
>> >
>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>> > that reboots the system by reading a.out into user memory (in the high
>> > 4K of core), then copies it to low memory and jumping to the entry
>> > point.  The name suggests its original intended use was to test a new
>> > system (kernel).
>> >
>> > P.S.
>> > Normal bootable system images seem to have been stored in reserved
>> > tracks of the (fixed head) disk (that are inacessible via system calls):
>> >
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>> > reads a.out and uses I/O instructions to write it out.
>> >
>> > P.P.S.
>> > Accordingly, I put together a "paper tape" for booting the system:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>> >
>> > P.P.P.S.
>> > The system (kernel) is 3K words, the last 1K of low memory
>> > used for the character table for the vector graphics controller.
>> >
>> > The definitions for the table are compiled by
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>> > from definition file
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>> > (after, ISTR, figuring out the ordering of the listing pages!)
>> >
>> > I don't think we ever figured out how the initial character table
>> > is loaded into core.  One thing that was missing from the table
>> > was the dispatch array, which I recreated:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>> >
>> > The system (kernel) could be built for a "cold start", reloading the
>> > disk (prone to head crashes?) from paper tape? But I don't think
>> > anyone ever reconstructed the procedure for rebuilding a disk that way.
>> >
>> > The disk was two sided, and the running system only used one side:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>> > appear to be programs to save and restore the filesystem from the
>> > "other" side of the disk.
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 6095 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Bakul Shah via TUHS]

[-- Attachment #1: Type: text/plain, Size: 3497 bytes --]

I would prefer old kernel to new kernel handoff if it can be made to work reliably. Nowadays there are a lot of things that run before the kernel gets control. 

> On Sep 18, 2024, at 3:38 PM, ron minnich <rminnich@gmail.com> wrote:
> 
> Interesting about the amiga. I'm assuming their firmware zeros memory on reset, so you have to do handoff from kernel to kernel, not via a reset and so on?
> 
> What was particularly nice about the V6/PDP-11 case: we were able to yank reset, which let us cleanly reset/disable devices, because everything was in memory when we got back. I miss the simplicity of the old machines.
> 
> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org <mailto:chopps@chopps.org>> wrote:
>> 
>> We had/have this functionality in the Amiga port of NetBSD.
>> 
>> It is implemented as `/dev/reload` device and you copy a kernel image to it. In locore.s there's code that copies the kernel image over top of the running kernel and then restarts. I believe for it to work nothing below the copy code in locore.s can change :)
>> 
>> Thanks,
>> Chris.
>> 
>> Phil Budne <phil@ultimate.com <mailto:phil@ultimate.com>> writes:
>> 
>> > ron minnich wrote:
>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
>> >> was there something before?
>> >
>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>> > that reboots the system by reading a.out into user memory (in the high
>> > 4K of core), then copies it to low memory and jumping to the entry
>> > point.  The name suggests its original intended use was to test a new
>> > system (kernel).
>> >
>> > P.S.
>> > Normal bootable system images seem to have been stored in reserved
>> > tracks of the (fixed head) disk (that are inacessible via system calls):
>> >
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>> > reads a.out and uses I/O instructions to write it out.
>> >
>> > P.P.S.
>> > Accordingly, I put together a "paper tape" for booting the system:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>> >
>> > P.P.P.S.
>> > The system (kernel) is 3K words, the last 1K of low memory
>> > used for the character table for the vector graphics controller.
>> >
>> > The definitions for the table are compiled by
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>> > from definition file
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>> > (after, ISTR, figuring out the ordering of the listing pages!)
>> >
>> > I don't think we ever figured out how the initial character table
>> > is loaded into core.  One thing that was missing from the table
>> > was the dispatch array, which I recreated:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>> >
>> > The system (kernel) could be built for a "cold start", reloading the
>> > disk (prone to head crashes?) from paper tape? But I don't think
>> > anyone ever reconstructed the procedure for rebuilding a disk that way.
>> >
>> > The disk was two sided, and the running system only used one side:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>> > appear to be programs to save and restore the filesystem from the
>> > "other" side of the disk.
>> 


[-- Attachment #2: Type: text/html, Size: 5281 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Peter Yardley]

I have memories of my colleague repairing that core memory with a large magnifier and a fine soldering iron.

> On 19 Sep 2024, at 8:52 AM, Luther Johnson <luther.johnson@makerlisp.com> wrote:
> 
> Even better were DG Novas with core memory, just turn the machine off, come back in the morning, turn on the machine, pick up where you left off :)
> On 09/18/2024 03:38 PM, ron minnich wrote:
>> Interesting about the amiga. I'm assuming their firmware zeros memory on reset, so you have to do handoff from kernel to kernel, not via a reset and so on? 
>> 
>> What was particularly nice about the V6/PDP-11 case: we were able to yank reset, which let us cleanly reset/disable devices, because everything was in memory when we got back. I miss the simplicity of the old machines.
>> 
>> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org> wrote:
>> 
>> We had/have this functionality in the Amiga port of NetBSD.
>> 
>> It is implemented as `/dev/reload` device and you copy a kernel image to it. In locore.s there's code that copies the kernel image over top of the running kernel and then restarts. I believe for it to work nothing below the copy code in locore.s can change :)
>> 
>> Thanks,
>> Chris.
>> 
>> Phil Budne <phil@ultimate.com> writes:
>> 
>> > ron minnich wrote:
>> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
>> >> was there something before?
>> >
>> > There was!  The PDP-7 UNIX listings contain a program trysys.s
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>> > that reboots the system by reading a.out into user memory (in the high
>> > 4K of core), then copies it to low memory and jumping to the entry
>> > point.  The name suggests its original intended use was to test a new
>> > system (kernel).
>> >
>> > P.S.
>> > Normal bootable system images seem to have been stored in reserved
>> > tracks of the (fixed head) disk (that are inacessible via system calls):
>> >
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>> > reads a.out and uses I/O instructions to write it out.
>> >
>> > P.P.S.
>> > Accordingly, I put together a "paper tape" for booting the system:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>> >
>> > P.P.P.S.
>> > The system (kernel) is 3K words, the last 1K of low memory
>> > used for the character table for the vector graphics controller.
>> >
>> > The definitions for the table are compiled by
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>> > from definition file
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>> > (after, ISTR, figuring out the ordering of the listing pages!)
>> >
>> > I don't think we ever figured out how the initial character table
>> > is loaded into core.  One thing that was missing from the table
>> > was the dispatch array, which I recreated:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>> >
>> > The system (kernel) could be built for a "cold start", reloading the
>> > disk (prone to head crashes?) from paper tape? But I don't think
>> > anyone ever reconstructed the procedure for rebuilding a disk that way.
>> >
>> > The disk was two sided, and the running system only used one side:
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>> > appear to be programs to save and restore the filesystem from the
>> > "other" side of the disk.
>> 
> 

.1.3.6.1.4.1.8852.4.2
Peter Yardley
peter.martin.yardley@gmail.com

[TUHS] Re: kernel boots kernel in 1977

[Luther Johnson]

[-- Attachment #1: Type: text/plain, Size: 3681 bytes --]

Even better were DG Novas with core memory, just turn the machine off,
come back in the morning, turn on the machine, pick up where you left off :)

On 09/18/2024 03:38 PM, ron minnich wrote:
> Interesting about the amiga. I'm assuming their firmware zeros memory
> on reset, so you have to do handoff from kernel to kernel, not via a
> reset and so on?
>
> What was particularly nice about the V6/PDP-11 case: we were able to
> yank reset, which let us cleanly reset/disable devices, because
> everything was in memory when we got back. I miss the simplicity of
> the old machines.
>
> On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org
> <mailto:chopps@chopps.org>> wrote:
>
>
>     We had/have this functionality in the Amiga port of NetBSD.
>
>     It is implemented as `/dev/reload` device and you copy a kernel
>     image to it. In locore.s there's code that copies the kernel image
>     over top of the running kernel and then restarts. I believe for it
>     to work nothing below the copy code in locore.s can change :)
>
>     Thanks,
>     Chris.
>
>     Phil Budne <phil@ultimate.com <mailto:phil@ultimate.com>> writes:
>
>     > ron minnich wrote:
>     >> But I'm wondering: is Ed's work in 1977 the first "kernel boots
>     kernel" or
>     >> was there something before?
>     >
>     > There was!  The PDP-7 UNIX listings contain a program trysys.s
>     > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
>     > that reboots the system by reading a.out into user memory (in
>     the high
>     > 4K of core), then copies it to low memory and jumping to the entry
>     > point.  The name suggests its original intended use was to test
>     a new
>     > system (kernel).
>     >
>     > P.S.
>     > Normal bootable system images seem to have been stored in reserved
>     > tracks of the (fixed head) disk (that are inacessible via system
>     calls):
>     >
>     > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
>     > reads a.out and uses I/O instructions to write it out.
>     >
>     > P.P.S.
>     > Accordingly, I put together a "paper tape" for booting the system:
>     >
>     https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>     >
>     > P.P.P.S.
>     > The system (kernel) is 3K words, the last 1K of low memory
>     > used for the character table for the vector graphics controller.
>     >
>     > The definitions for the table are compiled by
>     > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
>     > from definition file
>     > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
>     > (after, ISTR, figuring out the ordering of the listing pages!)
>     >
>     > I don't think we ever figured out how the initial character table
>     > is loaded into core.  One thing that was missing from the table
>     > was the dispatch array, which I recreated:
>     >
>     https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>     >
>     > The system (kernel) could be built for a "cold start", reloading the
>     > disk (prone to head crashes?) from paper tape? But I don't think
>     > anyone ever reconstructed the procedure for rebuilding a disk
>     that way.
>     >
>     > The disk was two sided, and the running system only used one side:
>     > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
>     > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
>     > appear to be programs to save and restore the filesystem from the
>     > "other" side of the disk.
>


[-- Attachment #2: Type: text/html, Size: 6650 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 3147 bytes --]

Interesting about the amiga. I'm assuming their firmware zeros memory on
reset, so you have to do handoff from kernel to kernel, not via a reset and
so on?

What was particularly nice about the V6/PDP-11 case: we were able to yank
reset, which let us cleanly reset/disable devices, because everything was
in memory when we got back. I miss the simplicity of the old machines.

On Wed, Sep 18, 2024 at 3:07 PM Christian Hopps <chopps@chopps.org> wrote:

>
> We had/have this functionality in the Amiga port of NetBSD.
>
> It is implemented as `/dev/reload` device and you copy a kernel image to
> it. In locore.s there's code that copies the kernel image over top of the
> running kernel and then restarts. I believe for it to work nothing below
> the copy code in locore.s can change :)
>
> Thanks,
> Chris.
>
> Phil Budne <phil@ultimate.com> writes:
>
> > ron minnich wrote:
> >> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel"
> or
> >> was there something before?
> >
> > There was!  The PDP-7 UNIX listings contain a program trysys.s
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
> > that reboots the system by reading a.out into user memory (in the high
> > 4K of core), then copies it to low memory and jumping to the entry
> > point.  The name suggests its original intended use was to test a new
> > system (kernel).
> >
> > P.S.
> > Normal bootable system images seem to have been stored in reserved
> > tracks of the (fixed head) disk (that are inacessible via system calls):
> >
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
> > reads a.out and uses I/O instructions to write it out.
> >
> > P.P.S.
> > Accordingly, I put together a "paper tape" for booting the system:
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
> >
> > P.P.P.S.
> > The system (kernel) is 3K words, the last 1K of low memory
> > used for the character table for the vector graphics controller.
> >
> > The definitions for the table are compiled by
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
> > from definition file
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
> > (after, ISTR, figuring out the ordering of the listing pages!)
> >
> > I don't think we ever figured out how the initial character table
> > is loaded into core.  One thing that was missing from the table
> > was the dispatch array, which I recreated:
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
> >
> > The system (kernel) could be built for a "cold start", reloading the
> > disk (prone to head crashes?) from paper tape? But I don't think
> > anyone ever reconstructed the procedure for rebuilding a disk that way.
> >
> > The disk was two sided, and the running system only used one side:
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
> > https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
> > appear to be programs to save and restore the filesystem from the
> > "other" side of the disk.
>
>

[-- Attachment #2: Type: text/html, Size: 4735 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Christian Hopps]

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]


We had/have this functionality in the Amiga port of NetBSD.

It is implemented as `/dev/reload` device and you copy a kernel image to it. In locore.s there's code that copies the kernel image over top of the running kernel and then restarts. I believe for it to work nothing below the copy code in locore.s can change :)

Thanks,
Chris.

Phil Budne <phil@ultimate.com> writes:

> ron minnich wrote:
>> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
>> was there something before?
>
> There was!  The PDP-7 UNIX listings contain a program trysys.s
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
> that reboots the system by reading a.out into user memory (in the high
> 4K of core), then copies it to low memory and jumping to the entry
> point.  The name suggests its original intended use was to test a new
> system (kernel).
>
> P.S.
> Normal bootable system images seem to have been stored in reserved
> tracks of the (fixed head) disk (that are inacessible via system calls):
>
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
> reads a.out and uses I/O instructions to write it out.
>
> P.P.S.
> Accordingly, I put together a "paper tape" for booting the system:
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>
> P.P.P.S.
> The system (kernel) is 3K words, the last 1K of low memory
> used for the character table for the vector graphics controller.
>
> The definitions for the table are compiled by
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
> from definition file
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
> (after, ISTR, figuring out the ordering of the listing pages!)
>
> I don't think we ever figured out how the initial character table
> is loaded into core.  One thing that was missing from the table
> was the dispatch array, which I recreated:
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>
> The system (kernel) could be built for a "cold start", reloading the
> disk (prone to head crashes?) from paper tape? But I don't think
> anyone ever reconstructed the procedure for rebuilding a disk that way.
>
> The disk was two sided, and the running system only used one side:
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
> appear to be programs to save and restore the filesystem from the
> "other" side of the disk.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 857 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 2295 bytes --]

Phil, that's really cool, thanks for it!

On Wed, Sep 18, 2024 at 8:34 AM Phil Budne <phil@ultimate.com> wrote:

> ron minnich wrote:
> > But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel"
> or
> > was there something before?
>
> There was!  The PDP-7 UNIX listings contain a program trysys.s
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
> that reboots the system by reading a.out into user memory (in the high
> 4K of core), then copies it to low memory and jumping to the entry
> point.  The name suggests its original intended use was to test a new
> system (kernel).
>
> P.S.
> Normal bootable system images seem to have been stored in reserved
> tracks of the (fixed head) disk (that are inacessible via system calls):
>
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
> reads a.out and uses I/O instructions to write it out.
>
> P.P.S.
> Accordingly, I put together a "paper tape" for booting the system:
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s
>
> P.P.P.S.
> The system (kernel) is 3K words, the last 1K of low memory
> used for the character table for the vector graphics controller.
>
> The definitions for the table are compiled by
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
> from definition file
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
> (after, ISTR, figuring out the ordering of the listing pages!)
>
> I don't think we ever figured out how the initial character table
> is loaded into core.  One thing that was missing from the table
> was the dispatch array, which I recreated:
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s
>
> The system (kernel) could be built for a "cold start", reloading the
> disk (prone to head crashes?) from paper tape? But I don't think
> anyone ever reconstructed the procedure for rebuilding a disk that way.
>
> The disk was two sided, and the running system only used one side:
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
> https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
> appear to be programs to save and restore the filesystem from the
> "other" side of the disk.
>

[-- Attachment #2: Type: text/html, Size: 3651 bytes --]

[TUHS] Re: kernel boots kernel in 1977

[Phil Budne]

ron minnich wrote:
> But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
> was there something before?

There was!  The PDP-7 UNIX listings contain a program trysys.s
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/trysys.s
that reboots the system by reading a.out into user memory (in the high
4K of core), then copies it to low memory and jumping to the entry
point.  The name suggests its original intended use was to test a new
system (kernel).

P.S.
Normal bootable system images seem to have been stored in reserved
tracks of the (fixed head) disk (that are inacessible via system calls):

https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/maksys.s
reads a.out and uses I/O instructions to write it out.

P.P.S.
Accordingly, I put together a "paper tape" for booting the system:
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/pbboot.s

P.P.P.S.
The system (kernel) is 3K words, the last 1K of low memory
used for the character table for the vector graphics controller.

The definitions for the table are compiled by
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/cas.s
from definition file
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/sys/cas.in
(after, ISTR, figuring out the ordering of the listing pages!)

I don't think we ever figured out how the initial character table
is loaded into core.  One thing that was missing from the table
was the dispatch array, which I recreated:
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/other/chrtbl.s

The system (kernel) could be built for a "cold start", reloading the
disk (prone to head crashes?) from paper tape? But I don't think
anyone ever reconstructed the procedure for rebuilding a disk that way.

The disk was two sided, and the running system only used one side:
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dsksav.s
https://github.com/DoctorWkt/pdp7-unix/blob/master/src/cmd/dskres.s
appear to be programs to save and restore the filesystem from the
"other" side of the disk.

[TUHS] Re: kernel boots kernel in 1977

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1793 bytes --]

On Wed, Sep 18, 2024, 2:47 PM ron minnich <rminnich@gmail.com> wrote:

> I noticed there are kexec talks this year at Linux Plumbers. Kexec, kernel
> boots kernel, has had a 25 year gestation in Linux, but it looks like it's
> finally coming together, driven by need.
>

Yes. LinuxBoot is fun... trick can be to slim the kernel + inird down
enough to fit in the NOR flash so it can be the second stage payload rather
than UEFI... i hacked FreeBSD's boot loader to be a Linux binary and use
kexec.

Thereby hangs a tale.
>
> in 1977, I was working at udel with Ed Szurkowski, the first sysadmin of
> the Unix systems we got in 1976 (first machine was an 11/70). Ed was a
> gifted engineer and did all kinds of neat work on v6. He later went to the
> Labs once he got his PhD and I lost track of him.
>
> Ed got tired of watching the bootstrap slowness, and wrote a system call
> that did the following:
> 1. load kernel in memory from system call argument
> 2. put special signature in low memory telling bootstrap "look in memory"
> 3. reboot via reset.
>
> Now, this works, because back then, ROM boot code did not zero memory.
> Memory was unchanged across reset. So the bootstrap could find the magic
> bits and start the kernel.
>
> I've lost the code, I carried the listings for decades but finally dumped
> them. A shame.
>

I saw similar code that would jump to location 0 to do the reset...  it
formed the basis of an interview question i used for two decades..

But I'm wondering: is Ed's work in 1977 the first "kernel boots kernel" or
> was there something before?
>

I've seen references to older mainframe systems doing that, but my quick
searches can't find a references to this, apart from the Chain command in
BASIC.

Warner

>

[-- Attachment #2: Type: text/html, Size: 2977 bytes --]