[COFF] [TUHS] man-page style

[COFF] [TUHS] man-page style

[jnc]

    > From: Grant Taylor

    >> as an operating system person, I was, and remain, a big fan of Multics
    >> ... which I still think was a better long-term path for OSes (long
    >> discussion of why elided to keep this short).

    > Can I ask for the longer discussion?

Sure. (Note that I'm not on COFF, so if anyone replies, please make sure I'm
CC'd directly.)


Basically, it boils down to 'monolithic OS's are bad' - for all the reasons
laid out in the usual ukernel places (I first saw them in the BSTJ MERT
paper, IIRC).

{What follows is not the crispest way to explain it; sorry. I didn't feel like
deleting it all and re-writing to get straight to the point, which is 'Multics
had many of the advantages of a ukernel - and a long time before the ukernel
idea arrived - but without the IPC overhead, which seems to be the biggest
argument against ukernels'.}

Now, early UNIX may have done pretty well in a _tiny_ amount of memory (I
think our -11/40 V6 had about 64KB of main memory _total_, or some such
ridiculous number), and to do that, maybe you have to go monolithic, but
monolithic is pretty Procrustean.

This was brought home to me in doing early TCP/IP work on PDP-11 Unix. The
system was just not well suited to networking work - if what you wanted to do
didn't fit into the UNIX model, you were screwed. Some people (e.g. BBN) did
TCP as a process (after adding IPC; no IPC in UNIX, back then), but then
you're talking a couple of process switches to do anything.

I looked at how Dave Clark was doing it on Multics, and I was green with envy.
He added, debugged and improved his code _on the running main campus system_,
sharing the machine with dozens of real users! Try doing that on UNIX
(although nowadays it's getting there, with loadable kernel stuff - but this
was in the 70's)!

To explain how this was even possible, you need to know a tiny bit about
Multics. It was a 'single level store' system; process memory and files were
not disjoint (as in UNIX), there were only segments (most of which were
long-lived, and survived reboots); processes had huge address spaces (up to
2^18 segments), and if you needed to use one, you just mapped it into your
address space, and off you went.

So he wrote his code, and if I in my command process executed the 'TELNET'
command, when it needed to open a TCP connection, it just mapped in his TCP
code, and called TCP$open() (or whatever he named it). It fiddled around in
the networking state (which was in a data segment that Dave had set up in his
'networking' process when it started up), and set things up. So it was kind of
doing a subroutine call to code in another process.

The security wasn't good, because Multics didn't have set-uid (so that only
Dave's code would have had access to that state database) - when they later
productized the code, they used Multics rings to make it secure.

But still, that was pretty amazing. The key thing here is that nobody had to
modify the Multics 'kernel' to support adding networking - the basic
mechanisms (the single-level store, etc) were so powerful and general you
could write entirely new basic things (like networking) and add them 'on the
fly'.


What Multics had was quite different from ukernels, but it amounted to the
same thing; the system wasn't this monolithic blob, it was
layered/modularized, and you could see (and gain access to, and in some cases
replace - either just for yourself, or for everyone) the layers/modules.

The nice thing was that to call up some subsystem to perform some service for
you, you didn't have to do IPC and then a process switch - it was a
_subroutine call_, in the CPU's hardware.

I don't think anyone else ever tried to go down that path as a way to
structure an operating system (in part because you need hardware support), and
it's a pity. (UNIX, which would run on anything, killed just about _everything
else_ off.)

The 386-Pentium actually had support for many segments, but I gather they are
in the process of deleting it in the latest machines because nobody's using
it. Which is a pity, because when done correctly (which it was - Intel hired
Paul Karger to architect it) it's just what you need for a truly secure system
(which Multics also had) - but that's another long message.

       Noel

[COFF] [TUHS] man-page style

[gtaylor]

On 11/17/2018 01:09 PM, Noel Chiappa wrote:
> Sure. (Note that I'm not on COFF, so if anyone replies, please make sure 
> I'm CC'd directly.)

Thank you for the reply Noel.  I found this to be an interesting read. 
I occasionally bump into some Multicians and am always pleasantly 
surprised at how different their world is.

> The 386-Pentium actually had support for many segments, but I gather 
> they are in the process of deleting it in the latest machines because 
> nobody's using it. Which is a pity, because when done correctly (which 
> it was - Intel hired Paul Karger to architect it) it's just what you 
> need for a truly secure system (which Multics also had) - but that's 
> another long message.

If you're ever feeling bored and would like to share that story.  ():-)

Thank you again for typing the message and sending it to a mailing list 
that you're not subscribed to.  If we're ever in a pub together, I owe 
you a round.



-- 
Grant. . . .
unix || die

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4008 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://minnie.tuhs.org/pipermail/coff/attachments/20181117/d5f094b1/attachment.bin>

[COFF] [TUHS] man-page style

[bakul]

KeyKOS had a lot in common with Multics. It had a ingle level store.
A KeyKOS system could stay on for a long time or can be restarted
from a snapshot. The kernel had no state of its own that needed
saving so snapshotting was easy. Unlike Multics' protection rings
it was a capability system so provided better security. KeyKOS
inter domain call was synchronous: the caller blocked until the
callee process was ready to accept the call. A later system EROS
was an evolution of KeyKOS and open source.

All of this would be easily possible on the Mill arch. if ever
it gets built. Mill has segments and protected function calls.

set-uid has its own issues. Plan9 doesn't have it.

On Nov 17, 2018, at 12:09 PM, Noel Chiappa <jnc at mercury.lcs.mit.edu> wrote:
> 
> To explain how this was even possible, you need to know a tiny bit about
> Multics. It was a 'single level store' system; process memory and files were
> not disjoint (as in UNIX), there were only segments (most of which were
> long-lived, and survived reboots); processes had huge address spaces (up to
> 2^18 segments), and if you needed to use one, you just mapped it into your
> address space, and off you went.
> 
> So he wrote his code, and if I in my command process executed the 'TELNET'
> command, when it needed to open a TCP connection, it just mapped in his TCP
> code, and called TCP$open() (or whatever he named it). It fiddled around in
> the networking state (which was in a data segment that Dave had set up in his
> 'networking' process when it started up), and set things up. So it was kind of
> doing a subroutine call to code in another process.
> 
> The security wasn't good, because Multics didn't have set-uid (so that only
> Dave's code would have had access to that state database) - when they later
> productized the code, they used Multics rings to make it secure.
> 
> But still, that was pretty amazing. The key thing here is that nobody had to
> modify the Multics 'kernel' to support adding networking - the basic
> mechanisms (the single-level store, etc) were so powerful and general you
> could write entirely new basic things (like networking) and add them 'on the
> fly'.
> 
> 
> What Multics had was quite different from ukernels, but it amounted to the
> same thing; the system wasn't this monolithic blob, it was
> layered/modularized, and you could see (and gain access to, and in some cases
> replace - either just for yourself, or for everyone) the layers/modules.
> 
> The nice thing was that to call up some subsystem to perform some service for
> you, you didn't have to do IPC and then a process switch - it was a
> _subroutine call_, in the CPU's hardware.
> 
> I don't think anyone else ever tried to go down that path as a way to
> structure an operating system (in part because you need hardware support), and
> it's a pity. (UNIX, which would run on anything, killed just about _everything
> else_ off.)
> 
> The 386-Pentium actually had support for many segments, but I gather they are
> in the process of deleting it in the latest machines because nobody's using
> it. Which is a pity, because when done correctly (which it was - Intel hired
> Paul Karger to architect it) it's just what you need for a truly secure system
> (which Multics also had) - but that's another long message.

[COFF] [TUHS] man-page style

[tytso]

On Sat, Nov 17, 2018 at 03:09:08PM -0500, Noel Chiappa wrote:
> 
> I looked at how Dave Clark was doing it on Multics, and I was green with envy.
> He added, debugged and improved his code _on the running main campus system_,
> sharing the machine with dozens of real users! Try doing that on UNIX
> (although nowadays it's getting there, with loadable kernel stuff - but this
> was in the 70's)!

One of the things that made Multics amazing is that you could replace
a shared library while other processes was using it --- and without
anything crashing.  To achieve this, there is a tag field which
identifies the object type and its length.  So if a library defines an
expanded version of the data structure, the new fields are tacked onto
the end of the data structure and the length field is bumped.  Older
callers of the library might pass in a version of the data structure
with the original length field; hence, fields can't be accessed unless
without first checking the structure tag.

I stole this idea and used in Kerberos v5 and Linux's userspace
ext2/3/4 utilities, where we use a error table code --- another
Multics concept --- as the structure tag.  So in the error_table file,
we might have:

ec	EXT2_ET_MAGIC_BADBLOCKS_ITERATE,
	"Wrong magic number for badblocks_iterate structure"

And that in each function that uses that structure, there'd be
something like this:

	EXT2_CHECK_MAGIC(iter, EXT2_ET_MAGIC_BADBLOCKS_ITERATE);

Where EXT2_CHECK_MAGIC is defined as:

#define EXT2_CHECK_MAGIC(struct, code) \
	  if ((struct)->magic != (code)) return (code)

(All MIT KerberosV5 and libext2fs structures have a 32-bit unsigned
magic field as the first 4 bytes of the structure.)

This technique is also useful so when I needed to add support for
64-bit block numbers, I could use the structure magic numbers to
disambiguate which version of the object we were using.  Hence unlike
some shared libraries where the magic number has been incremented to
indicate an ABI break every few months, e2fsprogs has not had an ABI
break in over ten years.

This also made it a bit easier to find use-after-free bugs in an era
before valgrid/purify, by the simple expedient of zeroing the magic
field when deallocating an object.

> The security wasn't good, because Multics didn't have set-uid (so that only
> Dave's code would have had access to that state database) - when they later
> productized the code, they used Multics rings to make it secure.

So that's a bit misleading.  Setuid isn't really a good analogue for
protection rings.  The proper analogue for user mode versus kernel
mode in the Unix world.  (Where user mode is roughly speaking, Multics
ring 4, and Kernel mode is Multics ring 0 --- the Honeywell hardware
had support for 8 rings, but processes running below ring 4 have so
little access that using them isn't terribly practical for general
purpose programs.  Processes ringing at rings 5 and higher wouldn't
have access to most of what we in the Unix world would call "the
standard POSIX system calls".)

Code running in one ring can transition to higher rings via "gates",
which would be the Unix equivalent of a system call.  Hence, a Multics
program running at Ring 4 could create its own gates that would
provide an extremeted limited set of system services to programs
running at Ring 5.  Those programs wouldn't have access to the normal
system calls, but only via the specified functions in the Ring 4
gates.  This is sort of like Capsicum, but it's more powerful --- and
it was designed decades before FreeBSD's Capsicum.

> The nice thing was that to call up some subsystem to perform some service for
> you, you didn't have to do IPC and then a process switch - it was a
> _subroutine call_, in the CPU's hardware.

Well, when you call a system call, you don't do a process switch
either.  So when Ring 4 code calls a ring 0 service, you chan think of
it as a system call.  It might not have been any slower than a normal
function call, but remember, this is a CISC system.  So another way of
saying things is that normal function calls weren't any faster than a
privilege transition via a system call!

> The 386-Pentium actually had support for many segments, but I gather they are
> in the process of deleting it in the latest machines because nobody's using
> it. Which is a pity, because when done correctly (which it was - Intel hired
> Paul Karger to architect it) it's just what you need for a truly secure system
> (which Multics also had) - but that's another long message.

One unfortunate thing about the 386 VM is that a segment plus offset
gets translated to a 32-bit global virtual address, which is then
translated to a physical address via a single page table.  With
Multics, each segment had its own page table which translated the
segment+offset to a physical address.

With only 32-bits of virtual address space on the 386, it's not at all
clear aggressive use of segments ala Multics would have worked
terribly well, due to the internal fragmentation of that 32-bit
address space.  So I've talked to some Multicians at MIT who might
quibble with the claim that 386's deisgn was "done correctly".

In any case, since no one really used segments on 32-bit x86, segment
support ended up getting mostly dropped in 64-bit mode.  (The FS and
GS segments still kinda work, mostly to keep Windows happy.  The CS,
DS, ES, and SS segments are basically no-ops in the 64-bit x86 world.)
Which is too bad.  I suspect that with a 64-bit address space,
designing an OS with a Multics-style segmentation architecture might
have been possible.

(But see Rob Pike's "Systems Software Research is Irrelevant" rant for
the argument that even if was *possible* it was very unlikely to have
happened, so for AMD and Intel to have neutered segmentation in the
x86-64 architecture might have been a well justified decision.)

Cheers,

					- Ted

[COFF] [TUHS] man-page style

[bakul]

On Mon, 19 Nov 2018 12:53:31 -0500 jnc at mercury.lcs.mit.edu (Noel Chiappa) wrote:
>
>     > All of this would be easily possible on the Mill arch. if ever it gets
>     > built. Mill has segments and protected function calls.
>
> What I found about that mostly talked about the belt stuff. Do you happen to
> have a pointer to the segment/call stuff?

This is a good talk re IPC and protection:

https://www.youtube.com/watch?v=XJasE5aOHSw

In the desciption below the video there is a
list of times where various topics are covered
so you can jump to what you want.

Slides here:
https://millcomputing.com/docs/inter-process-communication/

Ivan's talk on Security should also be of help:
https://www.youtube.com/watch?v=5osiYZV8n3U
https://millcomputing.com/docs/security/

The key implication is a thread can make a "portal" call,
where the same thread is now in a different protection domain.
No need for rendezvous & a couple of extra context switches to
a different thread, or trampoline through a higher privilege
kernel.  This callee function can only access what is visible
from its own protection domain. It can operate on caller's
memory data ony if the caller provides one time access to it.

>     > set-uid has its own issues. Plan9 doesn't have it.
>
> Ah, what were the issues (if you happen to know)?

The issue is setuid(uid,gid) process has *full* access*
available to uid,gid. If uid==0, now the process has superuser
access. Why should an install program have access to
/etc/passwd or have raw disk access or be able to root around
in kernel memory?  Typically you only want to provide very
limited access.

[COFF] [TUHS] man-page style

[jnc]

    > From: Grant Taylor

    > Thank you for the reply
    
Sure; I love to yap about stuff like this.

    > I occasionally bump into some Multicians and am always pleasantly
    > surprised at how different their world is.

Yes, it's very unlike most of what's been done since. Some of it (e.g. a
strictly ordered set of rings for protection) was wrong, but there's a lot of
good stuff there yet to be mined.

    >> Which is a pity, because when done correctly (which it was - Intel
    >> hired Paul Karger to architect it)

Ooops, it was Roger Schell, I get them mixed up all the time. His oral
history, here:

   https://conservancy.umn.edu/handle/11299/133439

is a Must Read.
  
    >> it's just what you > need for a truly secure system (which Multics also
    >> had) - but that's another long message.

    > If you're ever feeling bored and would like to share that story.

OK, soon.


    > From: Bakul Shah

    > All of this would be easily possible on the Mill arch. if ever it gets
    > built. Mill has segments and protected function calls.

What I found about that mostly talked about the belt stuff. Do you happen to
have a pointer to the segment/call stuff?

    > set-uid has its own issues. Plan9 doesn't have it.

Ah, what were the issues (if you happen to know)?

    Noel

[COFF] [TUHS] man-page style

[gtaylor]

On 11/16/2018 12:29 PM, Noel Chiappa wrote:
> _That_ is what made me such a huge fan of Unix, even though as an 
> operating system person, I was, and remain, a big fan of Multics (maybe 
> the only person in the world who really likes both :-), which I still 
> think was a better long-term path for OSes (long discussion of why elided 
> to keep this short).

Can I ask for the longer discussion?  It sounds like an enlightening 
sidebar that would be good to have over a cup of COFFee.  Maybe the 
barista on the COFF mailing list will brew you a cup to discuss this 
there.  ~wink~wink~nudge~nudge~



-- 
Grant. . . .
unix || die

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3982 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://minnie.tuhs.org/pipermail/coff/attachments/20181116/5dc2b58c/attachment.bin>