From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,HTML_MESSAGE,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 30135 invoked from network); 16 Sep 2021 19:28:30 -0000 Received: from minnie.tuhs.org (45.79.103.53) by inbox.vuxu.org with ESMTPUTF8; 16 Sep 2021 19:28:30 -0000 Received: by minnie.tuhs.org (Postfix, from userid 112) id B38009CABD; Fri, 17 Sep 2021 05:28:29 +1000 (AEST) Received: from minnie.tuhs.org (localhost [127.0.0.1]) by minnie.tuhs.org (Postfix) with ESMTP id 26EFC9CAB3; Fri, 17 Sep 2021 05:27:58 +1000 (AEST) Authentication-Results: minnie.tuhs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Yj69GlA0"; dkim-atps=neutral Received: by minnie.tuhs.org (Postfix, from userid 112) id 2CFE39CAB3; Fri, 17 Sep 2021 05:27:56 +1000 (AEST) Received: from mail-oi1-f173.google.com (mail-oi1-f173.google.com [209.85.167.173]) by minnie.tuhs.org (Postfix) with ESMTPS id 7B61A9CAB2 for ; Fri, 17 Sep 2021 05:27:54 +1000 (AEST) Received: by mail-oi1-f173.google.com with SMTP id s20so10571105oiw.3 for ; Thu, 16 Sep 2021 12:27:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uKbMl6s6HJ5qSffXwe+/wY7WdJ5t5sv+rZikxrpzntI=; b=Yj69GlA0xacGn8Dh/gpfkC5lfzlnQdBvKPknhfHHnN+X9nPbP4DcG9ZesTuEnA2nAF r6xndjh9eiWKA5vTF+M2xaAet4yZcu7T9dRsYBQiUJTSjlr74TvEK5W7qWBcjDR//B3A XGiKM34A3g9GifxQwJtqrU9Wbg6Ssl+7Afg1E4bGZEwqGl9F0yCWlOolqKrJg9xZREXZ r9x567u1dwG4uU4yJ2u68SVoEzDeNjEEqYPYtDhXmZvuM/JKNeZOo/3YVW17LGzARejK MmzOqtc9dXAhlshEUZPlk0vaWlq1xCDq37mRkaVd9/48kmryQ3v25CB4kkL2TMd2lwtN Yz/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uKbMl6s6HJ5qSffXwe+/wY7WdJ5t5sv+rZikxrpzntI=; b=sbt5sU61jLmA+qaklpHFVs80zYn5KcSCbFKyAwcjeI1fC9RnD1tkAEQ9BuYxWUi24d 1qx2hjdkHuRbMkSykR6pHYYTxo0aHlVKiuqYDbJ9N6IB4pHkpXWyAUsSefCg0cRHoKhW CrBk9hGmND7jL9MNa4t61jngYFE+MXuDnlFGkL2RxAXtxglShJ9nEVJ7nnvr9PWu/lWZ fARn5a/pmjHYS83PvLn3z+jQjlc7mgwnR6tRRaCx1ZMx+SvJgm9E46ZoYW8TajcfMrlN WbacPSYUuh2NSHwjaEk5gHLMEz6F25s9iaZbab1dK72lZ6A7TqzmG1ozxhUkVXy/ZrCI 0xBQ== X-Gm-Message-State: AOAM531qfVOBTHDnP0JrkjdN532HsGPFfVF1fTCE0f3wRjH3QvQCrKGX C4NJTHxVANy+WR9ueaIHHRe4oGDnueOHFpZ/hohm24k5Yj0= X-Google-Smtp-Source: ABdhPJzD5EX3gjeIkhB/ulH7JrqgHqybQb8pFSZs9yvAVC2sdo3hTxP5uKsYeOH8voJDIlonOY+R9vE17/pP8UOGRww= X-Received: by 2002:a05:6808:14c:: with SMTP id h12mr1174114oie.24.1631820473366; Thu, 16 Sep 2021 12:27:53 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dan Cross Date: Thu, 16 Sep 2021 15:27:17 -0400 Message-ID: To: "Theodore Ts'o" Content-Type: multipart/alternative; boundary="000000000000a6fd0705cc21cfa6" Subject: Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) X-BeenThere: tuhs@minnie.tuhs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: The Unix Heritage Society mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: TUHS main list , Douglas McIlroy Errors-To: tuhs-bounces@minnie.tuhs.org Sender: "TUHS" --000000000000a6fd0705cc21cfa6 Content-Type: text/plain; charset="UTF-8" On Fri, Sep 3, 2021 at 9:23 AM Theodore Ts'o wrote: > On Thu, Sep 02, 2021 at 11:24:37PM -0400, Douglas McIlroy wrote: > > I set out to write a reply, then found that Marshall had said it all, > > better..Alas, the crucial central principle of Plan 9 got ignored, while > > its ancillary contributions were absorbed into Linux, making Linux fatter > > but still oriented to a bygone milieu. > > I'm really not convinced trying to build distributed computing into > the OS ala Plan 9 is viable. It seems like plan9 itself is an existence proof that this is possible. What it did not present was an existence proof of its scalability and it wasn't successful commercially. It probably bears mentioning that that wasn't really the point of plan9, though; it was a research system. I'll try to address the plan9 specific bits below. The moment the OS has to span multiple > TCB's (Trusted Computing Bases), you have to make some very > opinionated decisions on a number of issues for which we do not have > consensus after decades of trial and error: > Interestingly, plan9 did make opinionated decisions about all of the things mentioned below. Largely, those decisions worked out pretty well. * What kind of directory service do you use? x.500/LDAP? Yellow Pages? > Project Athena's Hesiod? > In the plan9 world, the directory services were provided by the filesystem and a user-level library that consumed databases of information resident in the filesystem itself (ndb(6) -- https://9p.io/magic/man2html/6/ndb). It also provided DNS services for interacting with the larger world. The connection server was an idea that was ahead of it's time (ndb(8) -- https://9p.io/magic/man2html/8/ndb). Also, https://9p.io/sys/doc/net/net.html. * What kind of distributed authentication do you use? Kerboers? > Trust on first use authentication ala ssh? .rhosts style > "trust the network" style authentication? > Plan 9 specifically used a Kerberos-like model, but did not use the Kerberos protocol. I say "Kerberos-like" in that there was a trusted agent on the network that provided authentication services using a protocol based on shared secrets. * What kind of distributed authorization service do you use? Unix-style > numeric user-id/group-id's? X.500 Distinguished Names in ACL's? > Windows-style Security ID's? > User and group names were simple strings. There were no numeric UIDs associated with processes, though the original file server had a user<->integer mapping for directory entries. * Do you assume that all of the machines in your distributed > computation system belong to the same administrative domain? > Not necessarily, no. The model is one of resource sharing, rather than remote access. You usually pull resources into your local namespace, and those can come from anywhere you have access to take them from. They may come from a different "administrative domain". For example, towards the end of the BTL reign, there was a file server at Bell Labs that some folks had accounts on and that one could "import" into one's namespace. That was the main distribution point for the larger community. What if individuals owning their own workstations want to have > system administrator privs on their system? When a user logs into a Plan 9 terminal, they become the "host owner" of that terminal. The host owner is distinguished only in owning the hardware resources of the hosting machine; they have no other special privileges, nor is there an administrative user like `root`. It's slightly unusual, though not unheard of, for a terminal to have a local disk; the disk device is implicitly owned by the host owner. If the user puts a filesystem on that device (say, to cache a dataset locally or something), that's on them, though host owner status doesn't really give any special privileges over the permissions on the files on that filesystem, modulo them going through the raw disk device, of course. That is, the uid==0 implies you bypass all access permission checking is gone in Plan 9. CPU servers have a mechanism where a remote user can start a process on the server that becomes owned by the calling user; this is similar to remote login, except that the user brings their environment with them; the model is more of importing the CPU server's computational resources into the local environment than, say, ssh'ing into a machine. Or is your > distributed OS a niche system which only works when you have > clusters of machines that are all centrally and > administratively owned? > I'm not sure how to parse that; I mean, arguably networks of Unix machines associated with any given organization are "centrally owned and administered"? To get access to any given plan9 network, someone would have to create an account on the file and auth servers, but the system was also installable on a standalone machine with a local filesystem. If folks wanted to connect in from other types of systems, there were mechanisms for doing so: `ssh` and `telnet` servers were distributed and could be used, though the experience for an interactive user was pretty anemic. It was more typical to use a program called `drawterm` that runs as an application on e.g. a Mac or Unix machine and emulates enough of a Plan 9 terminal kernel that a user can effectively `cpu` to a plan9 CPU server. Once logged in via drawterm, one can run an environment including a window system and all the graphical stuff from there. Perhaps the aforementioned Bell Labs file server example clarifies things a bit? * What scale should the distributed system work at? 10's of machines > in a cluster? 100's of machines? 1000's of machines? > Tens of thousands of machines? This is, I think, where one gets to the crux of the problem. Plan 9 worked _really_ well for small clusters of machines (10s) and well enough for larger clusters (up to 100s or 1000s). Distributed systems that work > well on football-sized data centers may not work that well > when you only have a few racks in colo facility. The "I forgot > how to count that low" challenge is a real one... And how. Plan 9 _was_ eventually ported to football-field sized machines (the BlueGene port for DoE was on that scale); Ron can be able to speak to that in more detail, if he is so inclined. In fairness, I do think that required significant effort and it was, of course, highly specialized to HPC applications. My subjective impression was that any given plan9 network would break down at the scale of single-digit thousands of machines and, perhaps, tens of thousands of users. Growing beyond that for general use would probably require some pretty fundamental changes; for example, 9P (the file protocol) includes a client-chosen "FID" in transactions related to any open file, so that file servers must keep track of client state to associate fids to actual files, whether those file refer to disk-resident stable storage or software synthesized "files" for other things (IPC end points; process memory; whatever). There have been many, many proposals in the distributed computing > arena which all try to answer these questions differently. Solaris > had an answer with Yellow Pages, NFS, etc. OSF/DCE had an answer > involving Kerberos, DCE/RPC, DCE/DFS, etc. More recently we have > Docker's Swarm and Kubernetes, etc. None have achieved dominance, and > that should tell us something. > > The advantage of trying push all of these questions into the OS is > that you can try to provide the illusion that there is no difference > between local and remote resources. Is that the case, or is it that system designers try to provide uniform access to different classes of resources? Unix treats socket descriptors very similar to file descriptors very similar to pipes; why shouldn't named resources be handled in similar ways? But that either means that you > have a toy (sorry, "research") system which ignores all of the ways in > which remote computation which extends to a different node that may or > may not be up, which may or may not have belong to a different > administration domain, which may or may not have an adversary on the > network between you and the remote node, etc. OR, you have to make > access to local resources just as painful as access to remote > resources. Furthermore, since supporting access remote resources is > going to have more overhead, the illusion that access to local and > remote resources can be the same can't be comfortably sustained in any > case. > ...or some other way, which we'll never know about because no one thinks to ask the question, "how could we do this differently?" I think that's the crux of Mothy's point. Plan 9, as just one example, asked a lot of questions about the issues you mentioned above 30 years ago. They came up with _a_ set of answers; that set did evolve over time as things progressed. That doesn't mean that those questions were resolved definitively, just that there was a group of researchers who came up with an approach to them that worked for that group. What's changed is that we now take for granted that Linux is there, and we've stopped asking questions about anything outside of that model. When you add to that the complexities of building an OS that tries to > do a really good job supporting local resources --- see all of the > observations in Rob Pike's Systems Software Research is Dead slides > about why this is hard --- it seems to me the solution of trying to > build a hard dividing line between the Local OS and Distributed > Computation infrastructure is the right one. > > There is a huge difference between creating a local OS that can live > on a single developer's machine in their house --- and a distributed > OS which requires setting up a directory server, and an authentication > server, and a secure distributed time server, etc., before you set up > the first useful node that can actually run user workloads. You can > try to do both under a single source tree, but it's going to result in > a huge amount of bloat, and a huge amount of maintenance burden to > keep it all working. I agree with the first part of this paragraph, but then we're talking about researchers, not necessarily unaffiliated open-source developers. Hopefully researchers have some organizational and infrastructure support! By keeping the local node OS and the distributed computation system > separate, it can help control complexity, and that's a big part of > computer science, isn't it? > I don't know. It seems like this whole idea of distributed systems built on networks of loosely coupled, miniature timesharing systems has led to enormous amounts of inescapable complexity. I'll bet the Kubernetes by itself is larger than all of plan9. - Dan C. --000000000000a6fd0705cc21cfa6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Fri, Sep 3, 2021 at 9:23 AM Theodo= re Ts'o <tytso@mit.edu> wrot= e:
On Thu, Sep 0= 2, 2021 at 11:24:37PM -0400, Douglas McIlroy wrote:
> I set out to write a reply, then found that Marshall had said it all,<= br> > better..Alas, the crucial central principle of Plan 9 got ignored, whi= le
> its ancillary contributions were absorbed into Linux, making Linux fat= ter
> but still oriented to a bygone milieu.

I'm really not convinced trying to build distributed computing into
the OS ala Plan 9 is viable.

It seems like= plan9 itself is an existence proof that this is possible. What it did not = present was an existence proof of its scalability and it wasn't success= ful commercially. It probably bears mentioning that that wasn't really = the point of plan9, though; it was a research system.

<= div>I'll try to address the plan9 specific bits below.

The moment the OS has= to span multiple
TCB's (Trusted Computing Bases), you have to make some very
opinionated decisions on a number of issues for which we do not have
consensus after decades of trial and error:

=
Interestingly, plan9 did make opinionated decisions about all of the t= hings mentioned below. Largely, those decisions worked out pretty well.

=C2=A0 =C2=A0* What kind of directory service do you use?=C2=A0 x.500/LDAP?= =C2=A0 =C2=A0Yellow Pages?
=C2=A0 =C2=A0 =C2=A0 =C2=A0Project Athena's Hesiod?

In the plan9 world, the directory services were provided b= y the filesystem and a user-level library that consumed databases of inform= ation resident in the filesystem itself (ndb(6) -- https://9p.io/magic/man2html/6/ndb). It also pro= vided DNS services for interacting with the larger world. The connection se= rver was an idea that was ahead of it's time (ndb(8) -- https://9p.io/magic/man2html/8/ndb). Al= so,=C2=A0https://9p.io/sys/d= oc/net/net.html.

=C2=A0 =C2=A0* What kind of distributed authentication do you use?=C2=A0 Ke= rboers?
=C2=A0 =C2=A0 =C2=A0 =C2=A0Trust on first use authentication ala ssh?=C2=A0= .rhosts style
=C2=A0 =C2=A0 =C2=A0 =C2=A0"trust the network" style authenticati= on?

Plan 9 specifically used a Kerberos= -like model, but did not use the Kerberos protocol. I say "Kerberos-li= ke" in that there was a trusted agent on the network that provided aut= hentication services using a protocol based on shared secrets.
=C2=A0 =C2=A0* What kind of distributed authorization service do you use?= =C2=A0 =C2=A0Unix-style
=C2=A0 =C2=A0 =C2=A0 =C2=A0numeric user-id/group-id's?=C2=A0 =C2=A0X.50= 0 Distinguished Names in ACL's?
=C2=A0 =C2=A0 =C2=A0 =C2=A0Windows-style Security ID's?

User and group names were simple strings. There were n= o numeric UIDs associated with processes, though the original file server h= ad a user<->integer mapping for directory entries.

=C2=A0 =C2=A0* Do you assume that all of the machines in your distributed =C2=A0 =C2=A0 =C2=A0 =C2=A0computation system belong to the same administra= tive domain?

Not necessarily, no. The m= odel is one of resource sharing, rather than remote access. You usually pul= l resources into your local namespace, and those can come from anywhere you= have access to take them from. They may come from a different "admini= strative domain". For example, towards the end of the BTL reign, there= was a file server at Bell Labs that some folks had accounts on and that on= e could "import" into one's namespace. That was the main dist= ribution point for the larger community.

=C2=A0 =C2=A0 =C2=A0 =C2=A0What if individuals owning their own workstation= s want to have
=C2=A0 =C2=A0 =C2=A0 =C2=A0system administrator privs on their system?

When a user logs into a Plan 9 terminal, they b= ecome the "host owner" of that terminal. The host owner is distin= guished only in owning the hardware resources of the hosting machine; they = have no other special privileges, nor is there an administrative user like = `root`. It's slightly unusual, though not unheard of, for a terminal to= have a local disk; the disk device is implicitly owned by the host owner. = If the user puts a filesystem on that device (say, to cache a dataset local= ly or something), that's on them, though host owner status doesn't = really give any special privileges over the permissions on the files on tha= t filesystem, modulo them going through the raw disk device, of course. Tha= t is, the uid=3D=3D0 implies you bypass all access permission checking is g= one in Plan 9. CPU servers have a mechanism where a remote user can start a= process on the server that becomes owned by the calling user; this is simi= lar to remote login, except that the user brings their environment with the= m; the model is more of importing the CPU server's computational resour= ces into the local environment than, say, ssh'ing into a machine.
=

Or is your=
=C2=A0 =C2=A0 =C2=A0 =C2=A0distributed OS a niche system which only works w= hen you have
=C2=A0 =C2=A0 =C2=A0 =C2=A0clusters of machines that are all centrally and<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0administratively owned?
I'm not sure how to parse that; I mean, arguably networks o= f Unix machines associated with any given organization are "centrally = owned and administered"? To get access to any given plan9 network, som= eone would have to create an account on the file and auth servers, but the = system was also installable on a standalone machine with a local filesystem= . If folks wanted to connect in from other types of systems, there were mec= hanisms for doing so: `ssh` and `telnet` servers were distributed and could= be used, though the experience for an interactive user was pretty anemic. = It was more typical to use a program called `drawterm` that runs as an appl= ication on e.g. a Mac or Unix machine and emulates enough of a Plan 9 termi= nal kernel that a user can effectively `cpu` to a plan9 CPU server. Once lo= gged in via drawterm, one can run an environment including a window system = and all the graphical stuff from there.

Perhaps th= e aforementioned Bell Labs file server example clarifies things a bit?
<= /div>

=C2=A0 =C2=A0* What scale should the distributed system work at?=C2=A0 10&#= 39;s of machines
=C2=A0 =C2=A0 =C2=A0 =C2=A0in a cluster?=C2=A0 =C2=A0100's of machines?= =C2=A0 1000's of machines?
=C2=A0 =C2=A0 =C2=A0 =C2=A0Tens of thousands of machines?
=
This is, I think, where one gets to the crux of the problem.= Plan 9 worked _really_ well for small clusters of machines (10s) and well = enough for larger clusters (up to 100s or 1000s).

Distributed systems that work<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0well on football-sized data centers may not work= that well
=C2=A0 =C2=A0 =C2=A0 =C2=A0when you only have a few racks in colo facility.= =C2=A0 The "I forgot
=C2=A0 =C2=A0 =C2=A0 =C2=A0how to count that low" challenge is a real = one...

And how. Plan 9 _was_ eventually por= ted to football-field sized machines (the BlueGene port for DoE was on that= scale); Ron can be able to speak to that in more detail, if he is so incli= ned. In fairness, I do think that required significant effort and it was, o= f course, highly specialized to HPC applications.

= My subjective impression was that any given plan9 network would break down = at the scale of single-digit thousands of machines and, perhaps, tens of th= ousands of users. Growing beyond that for general use would probably requir= e some pretty fundamental changes; for example, 9P (the file protocol) incl= udes a client-chosen "FID" in transactions related to any open fi= le, so that file servers=C2=A0must keep track of client state to associate = fids to actual files, whether those file refer to disk-resident stable stor= age or software synthesized "files" for other things (IPC end poi= nts; process memory; whatever).

There have been many, many proposals in the distributed computing
arena which all try to answer these questions differently.=C2=A0 Solaris had an answer with Yellow Pages, NFS, etc.=C2=A0 OSF/DCE had an answer
involving Kerberos, DCE/RPC, DCE/DFS, etc.=C2=A0 More recently we have
Docker's Swarm and Kubernetes, etc.=C2=A0 None have achieved dominance,= and
that should tell us something.

The advantage of trying push all of these questions into the OS is
that you can try to provide the illusion that there is no difference
between local and remote resources.

Is that= the case, or is it that system designers try to provide uniform access to = different classes of resources? Unix treats socket descriptors very similar= to file descriptors very similar to pipes; why shouldn't named resourc= es be handled in similar ways?

But that either means that you
have a toy (sorry, "research") system which ignores all of the wa= ys in
which remote computation which extends to a different node that may or
may not be up, which may or may not have belong to a different
administration domain, which may or may not have an adversary on the
network between you and the remote node, etc.=C2=A0 OR, you have to make access to local resources just as painful as access to remote
resources.=C2=A0 Furthermore, since supporting access remote resources is going to have more overhead, the illusion that access to local and
remote resources can be the same can't be comfortably sustained in any<= br> case.

...or some other way, which we= 9;ll never know about because no one thinks to ask the question, "how = could we do this differently?" I think that's the crux of Mothy= 9;s point.

Plan 9, as just one example, asked a lo= t of questions about the issues you mentioned above 30 years ago. They came= up with _a_ set of answers; that set did evolve over time as things progre= ssed. That doesn't mean that those questions were resolved definitively= , just that there was a group of researchers who came up with an approach t= o them that worked for that group.

What's chan= ged is that we now take for granted that Linux is there, and we've stop= ped asking questions about anything outside of that model.

When you add to that the complexities of building an OS that tries to
do a really good job supporting local resources --- see all of the
observations in Rob Pike's Systems Software Research is Dead slides
about why this is hard --- it seems to me the solution of trying to
build a hard dividing line between the Local OS and Distributed
Computation infrastructure is the right one.

There is a huge difference between creating a local OS that can live
on a single developer's machine in their house --- and a distributed OS which requires setting up a directory server, and an authentication
server, and a secure distributed time server, etc., before you set up
the first useful node that can actually run user workloads.=C2=A0 You can try to do both under a single source tree, but it's going to result in<= br> a huge amount of bloat, and a huge amount of maintenance burden to
keep it all working.

I agree with the first= part of this paragraph, but then we're talking about researchers, not = necessarily unaffiliated open-source developers. Hopefully researchers have= some organizational and infrastructure support!

By keeping the local node OS and the distributed computation system
separate, it can help control complexity, and that's a big part of
computer science, isn't it?

I don&#= 39;t know. It seems like this whole idea of distributed systems built on ne= tworks of loosely coupled, miniature timesharing systems has led to enormou= s amounts of inescapable complexity.

I'll bet = the Kubernetes by itself is larger than all of plan9.

<= div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 - Dan C.

--000000000000a6fd0705cc21cfa6--