The Unix Heritage Society mailing list
 help / color / mirror / Atom feed
* [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
@ 2021-09-01 21:58 Dan Cross
  2021-09-02  8:42 ` Tony Finch
                   ` (3 more replies)
  0 siblings, 4 replies; 52+ messages in thread
From: Dan Cross @ 2021-09-01 21:58 UTC (permalink / raw)
  To: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 3676 bytes --]

One of the things I really appreciate about participating in this community
and studying Unix history (and the history of other systems) is that it
gives one firm intellectual ground from which to evaluate where one is
going: without understanding where one is and where one has been, it's
difficult to assert that one isn't going sideways or completely backwards.
Maybe either of those outcomes is appropriate at times (paradigms shift; we
make mistakes; etc) but generally we want to be moving mostly forward.

The danger when immersing ourselves in history, where we must consider and
appreciate the set of problems that created the evolutionary paths leading
to the systems we are studying, is that our thinking can become calcified
in assuming that those systems continue to meet the needs of the problems
of today. It is therefore always important to reevaluate our base
assumptions in light of either disconfirming evidence or (in our specific
case) changing environments.

To that end, I found Timothy Roscoe's (ETH) joint keynote address at
ATC/OSDI'21 particularly compelling. He argues that what we consider the
"operating system" is only controlling a fraction of a modern computer
these days, and that in many ways our models for what we consider "the
computer" are outdated and incomplete, resulting in systems that are
artificially constrained, insecure, and with separate components that do
not consider each other and therefore frequently conflict. Further,
hardware is ossifying around the need to present a system interface that
can be controlled by something like Linux (used as a proxy more generally
for a Unix-like operating system), simultaneously broadening the divide and
making it ever more entrenched.

Another theme in the presentation is that, to the limited extent
the broader systems research community is actually approaching OS topics at
all, it is focusing almost exclusively on Linux in lieu of new, novel
systems; where non-Linux systems are featured (something like 3 accepted
papers between SOSP and OSDI in the last two years out of $n$), the
described systems are largely Linux-like. Here the presentation reminded me
of Rob Pike's "Systems Software Research is Irrelevant" talk (slides of
which are available in various places, though I know of no recording of
that talk).

Roscoe's challenge is that all of this should be seen as both a challenge
and an opportunity for new research into operating systems specifically:
what would it look like to take a holistic approach towards the hardware
when architecting a new system to drive all this hardware? We have new
tools that can make this tractable, so why don't we do it? Part of it is
bias, but part of it is that we've lost sight of the larger picture. My own
question is, have we become entrenched in the world of systems that are
"good enough"?

Things he does NOT mention are system interfaces to userspace software; he
doesn't seem to have any quibbles with, say, the Linux system call
interface, the process model, etc. He's mostly talking about taking into
account the hardware. Also, in fairness, his highlighting a "small" portion
of the system and saying, "that's what the OS drives!" sort of reminds me
of the US voter maps that show vast tracts of largely unpopulated land
colored a certain shade as having voted for a particular candidate, without
normalizing for population (land doesn't vote, people do, though in the US
there is a relationship between how these things impact the overall
election for, say, the presidency).

I'm curious about other peoples' thoughts on the talk and the overall topic?

https://www.youtube.com/watch?v=36myc8wQhLo

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 4090 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-01 21:58 [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
@ 2021-09-02  8:42 ` Tony Finch
  2021-09-03  0:19   ` John Cowan
  2021-09-02 15:41 ` Kevin Bowling
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 52+ messages in thread
From: Tony Finch @ 2021-09-02  8:42 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 1583 bytes --]

Dan Cross <crossd@gmail.com> wrote:
>
> I'm curious about other peoples' thoughts on the talk and the overall topic?

I saw this talk yesterday and I twote some off-the-cuff thoughts
(https://twitter.com/fanf/status/1433004863818960898):

I wanted to hear more about the problem of closed firmware (Roscoe's
research platform uses OpenBMC). I hope the gradual rise of open firmware
will help to fix the problem of platform management controllers treating
the OS as the enemy.

The talk focuses on Linux because ~everything runs Linux, but Linux was
designed for a platform that was defined by Intel and Microsoft, and the
disfunctional split that Roscoe points out is exactly the split in design
responsibilities between Intel and Microsoft. In the Arm world the split
has been reproduced, but with Arm and the SoC manufacturers instead of
Intel and the PC clones, and Linux instead of Windows.

There’s also Conway’s law, “you ship your org chart”, and in this case
Roscoe is talking about the organisation of the computer industry as a
whole. So if someone comes up with a better OS architecture, is the
implied org chart also successful under capitalism?

(end paste)

I suppose the historical perspective would be to ask if the way that OS
and driver software was developed in the past in vertically-integratewd
companies can provide insight into the hardware complexity of today's
systems...

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  https://dotat.at/
Bailey: South or southwest, becoming variable, 2 to 4. Slight.
Showers. Good.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-01 21:58 [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
  2021-09-02  8:42 ` Tony Finch
@ 2021-09-02 15:41 ` Kevin Bowling
  2021-09-02 20:12   ` Marshall Conover
  2021-09-03 15:56 ` Warner Losh
  2021-09-16 18:38 ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
  3 siblings, 1 reply; 52+ messages in thread
From: Kevin Bowling @ 2021-09-02 15:41 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list

On Wed, Sep 1, 2021 at 3:00 PM Dan Cross <crossd@gmail.com> wrote:
>
> One of the things I really appreciate about participating in this community and studying Unix history (and the history of other systems) is that it gives one firm intellectual ground from which to evaluate where one is going: without understanding where one is and where one has been, it's difficult to assert that one isn't going sideways or completely backwards. Maybe either of those outcomes is appropriate at times (paradigms shift; we make mistakes; etc) but generally we want to be moving mostly forward.
>
> The danger when immersing ourselves in history, where we must consider and appreciate the set of problems that created the evolutionary paths leading to the systems we are studying, is that our thinking can become calcified in assuming that those systems continue to meet the needs of the problems of today. It is therefore always important to reevaluate our base assumptions in light of either disconfirming evidence or (in our specific case) changing environments.
>
> To that end, I found Timothy Roscoe's (ETH) joint keynote address at ATC/OSDI'21 particularly compelling. He argues that what we consider the "operating system" is only controlling a fraction of a modern computer these days, and that in many ways our models for what we consider "the computer" are outdated and incomplete, resulting in systems that are artificially constrained, insecure, and with separate components that do not consider each other and therefore frequently conflict. Further, hardware is ossifying around the need to present a system interface that can be controlled by something like Linux (used as a proxy more generally for a Unix-like operating system), simultaneously broadening the divide and making it ever more entrenched.
>
> Another theme in the presentation is that, to the limited extent the broader systems research community is actually approaching OS topics at all, it is focusing almost exclusively on Linux in lieu of new, novel systems; where non-Linux systems are featured (something like 3 accepted papers between SOSP and OSDI in the last two years out of $n$), the described systems are largely Linux-like. Here the presentation reminded me of Rob Pike's "Systems Software Research is Irrelevant" talk (slides of which are available in various places, though I know of no recording of that talk).
>
> Roscoe's challenge is that all of this should be seen as both a challenge and an opportunity for new research into operating systems specifically: what would it look like to take a holistic approach towards the hardware when architecting a new system to drive all this hardware? We have new tools that can make this tractable, so why don't we do it? Part of it is bias, but part of it is that we've lost sight of the larger picture. My own question is, have we become entrenched in the world of systems that are "good enough"?
>
> Things he does NOT mention are system interfaces to userspace software; he doesn't seem to have any quibbles with, say, the Linux system call interface, the process model, etc. He's mostly talking about taking into account the hardware. Also, in fairness, his highlighting a "small" portion of the system and saying, "that's what the OS drives!" sort of reminds me of the US voter maps that show vast tracts of largely unpopulated land colored a certain shade as having voted for a particular candidate, without normalizing for population (land doesn't vote, people do, though in the US there is a relationship between how these things impact the overall election for, say, the presidency).
>
> I'm curious about other peoples' thoughts on the talk and the overall topic?
>
> https://www.youtube.com/watch?v=36myc8wQhLo
>
>         - Dan C.


One thing I've realized as the unit of computing becomes more and more
abundant (one off
HW->mainframes->minis->micros->servers->VMs->containers) the OS
increasingly becomes less visible and other software components become
more important.  It's an implementation detail like a language runtime
and software developers are increasingly ill equipped to work at this
layer.  Public cloud/*aaS is a major blow to interesting general
purpose OS work in commercial computing since businesses increasingly
outsource more and more of their workloads. The embedded (which
includes phones/Fuschia, accelerator firmware/payload, RTOS etc) and
academic (i.e. Cambridge CHERI) world may have to sustain OS research
for the foreseeable future.

There is plenty of systems work going on but it takes place in
different ways, userspace systems are completely viable and do not
require switching to microkernels.  Intel's DPDK/SPDK as one
ecosystem, Kubernetes as another - there is a ton of rich systems work
in this ecosystem with eBPF/XDP etc, and I used to dismiss it but it
is no longer possible to do so rationally.  I would go as far as
saying Kubernetes is _the_ datacenter OS and has subsumed Linux itself
as the primary system abstraction for the next while.. even Microsoft
has a native implementation on Server 2022.  It looks different and
smells different, but being able to program compute/storage/network
fabric with one abstraction is the holy grail of cluster computing and
interestingly it lets you swap the lower layer implementations out
with less risk but also less fanfare.

Regards,
Kevin

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-02 15:41 ` Kevin Bowling
@ 2021-09-02 20:12   ` Marshall Conover
  0 siblings, 0 replies; 52+ messages in thread
From: Marshall Conover @ 2021-09-02 20:12 UTC (permalink / raw)
  To: Kevin Bowling; +Cc: TUHS main list

Kevin, I think that's a great framing of why this talk actually seemed
inverted in its focus for me, and a good identification of why the
presenter might see OS development stalling out and ossifying around
Linux.

I come from the opposite side of the presenter here: my frustration as
a backend dev and user has been that modern OSs still think presenting
an abstraction over my resources means making it easy to use one
single machine (or, as the presenter brings up, a subset of the
machine). Instead, my resources are spread out among many machines and
a number of remote web services, for which I'd like to have one
seamless interface - both for development and use. From an OS
perspective, Plan 9 and its daughter systems have come the closest
I've seen to addressing this by intentionally thinking about the
problem and creating an API system for representing resources that
reaches across networks, and a mutable namespace for using and
manipulating those APIs. Despite pulling other ideas from 9, the
importance of having an opinion on the distributed nature of modern
computing seems to have been missed by prominent operating systems
today. As a result, their development has been relegated to what they
do: be a platform for things that actually provide an abstraction for
my resources.

And userspace systems have filled the demand for abstracting
distributed resource usage to demonstrable business success, if
questionable architectural success (as in, they can still be a
confusing pain in the buns and require excess work sometimes). As a
dev, the systems that have come the closest to presenting one unified
abstraction over my resources are the meta-services offered by Google,
MS and Amazon such as Azure and AWS.

I think the distributed nature of things today is also potentially why
the focus of the conference is on distributed systems now, as lamented
by the presenter. Granted that I'm not the sharpest bulb in the
drawer, but I can't think of a way an OS taking more direct control of
the internal hardware of an individual computer would impact me beyond
the security issues mentioned in the talk. However, I can think of a
number of ways an OS being opinionated about working with networked
machines would greatly improve my situation. Boy, it would be great to
just spin up a cluster of machines, install one OS on all of them, and
treat them as one resource. That's the dream the k8s mentality
promises, and MS and Amazon are already walking towards being this
sort of one-stop shop: "want cluster computing? Press a button to spin
up a cluster with ECS, and store your containers in ECR. Want to run a
program or twelve somewhere on the cluster? Just tell us which one and
how many. Worried about storage? Just tell us what size storage it
needs. We've got you covered!" None of it is perfect, but it shows
that there's heavy demand for a system where users don't have to think
about how to architect and maintain arbitrary groupings of their
resources as necessitated by how OSs think of their job now, and
instead just want to feel as if they're writing and running programs
on one big 'thing'.

So I think the ossification around Linux mentioned in the talk might
be that unless operating systems start doing something more than being
a host for the tools that actually provide an abstraction over all my
resources, there's no real reason to make them do anything else. If
you're not making it easier to use my resources than k8s or Azure, why
would I want you?

Cheers,

Marshall






On Thu, Sep 2, 2021 at 11:42 AM Kevin Bowling <kevin.bowling@kev009.com> wrote:
>
> On Wed, Sep 1, 2021 at 3:00 PM Dan Cross <crossd@gmail.com> wrote:
> >
> > One of the things I really appreciate about participating in this community and studying Unix history (and the history of other systems) is that it gives one firm intellectual ground from which to evaluate where one is going: without understanding where one is and where one has been, it's difficult to assert that one isn't going sideways or completely backwards. Maybe either of those outcomes is appropriate at times (paradigms shift; we make mistakes; etc) but generally we want to be moving mostly forward.
> >
> > The danger when immersing ourselves in history, where we must consider and appreciate the set of problems that created the evolutionary paths leading to the systems we are studying, is that our thinking can become calcified in assuming that those systems continue to meet the needs of the problems of today. It is therefore always important to reevaluate our base assumptions in light of either disconfirming evidence or (in our specific case) changing environments.
> >
> > To that end, I found Timothy Roscoe's (ETH) joint keynote address at ATC/OSDI'21 particularly compelling. He argues that what we consider the "operating system" is only controlling a fraction of a modern computer these days, and that in many ways our models for what we consider "the computer" are outdated and incomplete, resulting in systems that are artificially constrained, insecure, and with separate components that do not consider each other and therefore frequently conflict. Further, hardware is ossifying around the need to present a system interface that can be controlled by something like Linux (used as a proxy more generally for a Unix-like operating system), simultaneously broadening the divide and making it ever more entrenched.
> >
> > Another theme in the presentation is that, to the limited extent the broader systems research community is actually approaching OS topics at all, it is focusing almost exclusively on Linux in lieu of new, novel systems; where non-Linux systems are featured (something like 3 accepted papers between SOSP and OSDI in the last two years out of $n$), the described systems are largely Linux-like. Here the presentation reminded me of Rob Pike's "Systems Software Research is Irrelevant" talk (slides of which are available in various places, though I know of no recording of that talk).
> >
> > Roscoe's challenge is that all of this should be seen as both a challenge and an opportunity for new research into operating systems specifically: what would it look like to take a holistic approach towards the hardware when architecting a new system to drive all this hardware? We have new tools that can make this tractable, so why don't we do it? Part of it is bias, but part of it is that we've lost sight of the larger picture. My own question is, have we become entrenched in the world of systems that are "good enough"?
> >
> > Things he does NOT mention are system interfaces to userspace software; he doesn't seem to have any quibbles with, say, the Linux system call interface, the process model, etc. He's mostly talking about taking into account the hardware. Also, in fairness, his highlighting a "small" portion of the system and saying, "that's what the OS drives!" sort of reminds me of the US voter maps that show vast tracts of largely unpopulated land colored a certain shade as having voted for a particular candidate, without normalizing for population (land doesn't vote, people do, though in the US there is a relationship between how these things impact the overall election for, say, the presidency).
> >
> > I'm curious about other peoples' thoughts on the talk and the overall topic?
> >
> > https://www.youtube.com/watch?v=36myc8wQhLo
> >
> >         - Dan C.
>
>
> One thing I've realized as the unit of computing becomes more and more
> abundant (one off
> HW->mainframes->minis->micros->servers->VMs->containers) the OS
> increasingly becomes less visible and other software components become
> more important.  It's an implementation detail like a language runtime
> and software developers are increasingly ill equipped to work at this
> layer.  Public cloud/*aaS is a major blow to interesting general
> purpose OS work in commercial computing since businesses increasingly
> outsource more and more of their workloads. The embedded (which
> includes phones/Fuschia, accelerator firmware/payload, RTOS etc) and
> academic (i.e. Cambridge CHERI) world may have to sustain OS research
> for the foreseeable future.
>
> There is plenty of systems work going on but it takes place in
> different ways, userspace systems are completely viable and do not
> require switching to microkernels.  Intel's DPDK/SPDK as one
> ecosystem, Kubernetes as another - there is a ton of rich systems work
> in this ecosystem with eBPF/XDP etc, and I used to dismiss it but it
> is no longer possible to do so rationally.  I would go as far as
> saying Kubernetes is _the_ datacenter OS and has subsumed Linux itself
> as the primary system abstraction for the next while.. even Microsoft
> has a native implementation on Server 2022.  It looks different and
> smells different, but being able to program compute/storage/network
> fabric with one abstraction is the holy grail of cluster computing and
> interestingly it lets you swap the lower layer implementations out
> with less risk but also less fanfare.
>
> Regards,
> Kevin

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-02  8:42 ` Tony Finch
@ 2021-09-03  0:19   ` John Cowan
  2021-09-03  3:24     ` Douglas McIlroy
  0 siblings, 1 reply; 52+ messages in thread
From: John Cowan @ 2021-09-03  0:19 UTC (permalink / raw)
  To: Tony Finch; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

On Thu, Sep 2, 2021 at 5:01 AM Tony Finch <dot@dotat.at> wrote:


> Linux was
> designed for a platform that was defined by Intel and Microsoft, and the
> disfunctional split that Roscoe points out is exactly the split in design
> responsibilities between Intel and Microsoft.
>

As the saying has it: "What Andy giveth, Bill taketh away."

[-- Attachment #2: Type: text/html, Size: 876 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03  0:19   ` John Cowan
@ 2021-09-03  3:24     ` Douglas McIlroy
  2021-09-03 13:21       ` Theodore Ts'o
  0 siblings, 1 reply; 52+ messages in thread
From: Douglas McIlroy @ 2021-09-03  3:24 UTC (permalink / raw)
  To: John Cowan; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 878 bytes --]

I set out to write a reply, then found that Marshall had said it all,
better..Alas, the crucial central principle of Plan 9 got ignored, while
its ancillary contributions were absorbed into Linux, making Linux fatter
but still oriented to a bygone milieu.

Another entrant in the distributable computing arena is Jay Misra's
"orchestrator", Orc, in which practice goes hand-in-hand with theory:
https://www.cs.utexas.edu/users/misra/OrcBook.pdf.

Doug

On Thu, Sep 2, 2021 at 8:19 PM John Cowan <cowan@ccil.org> wrote:

>
>
> On Thu, Sep 2, 2021 at 5:01 AM Tony Finch <dot@dotat.at> wrote:
>
>
>> Linux was
>> designed for a platform that was defined by Intel and Microsoft, and the
>> disfunctional split that Roscoe points out is exactly the split in design
>> responsibilities between Intel and Microsoft.
>>
>
> As the saying has it: "What Andy giveth, Bill taketh away."
>

[-- Attachment #2: Type: text/html, Size: 2171 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03  3:24     ` Douglas McIlroy
@ 2021-09-03 13:21       ` Theodore Ts'o
  2021-09-08 11:14         ` Tony Finch
  2021-09-16 19:27         ` Dan Cross
  0 siblings, 2 replies; 52+ messages in thread
From: Theodore Ts'o @ 2021-09-03 13:21 UTC (permalink / raw)
  To: Douglas McIlroy; +Cc: TUHS main list

On Thu, Sep 02, 2021 at 11:24:37PM -0400, Douglas McIlroy wrote:
> I set out to write a reply, then found that Marshall had said it all,
> better..Alas, the crucial central principle of Plan 9 got ignored, while
> its ancillary contributions were absorbed into Linux, making Linux fatter
> but still oriented to a bygone milieu.

I'm really not convinced trying to build distributed computing into
the OS ala Plan 9 is viable.  The moment the OS has to span multiple
TCB's (Trusted Computing Bases), you have to make some very
opinionated decisions on a number of issues for which we do not have
consensus after decades of trial and error:

   * What kind of directory service do you use?  x.500/LDAP?   Yellow Pages?
       Project Athena's Hesiod?
   * What kind of distributed authentication do you use?  Kerboers?
       Trust on first use authentication ala ssh?  .rhosts style
       "trust the network" style authentication?
   * What kind of distributed authorization service do you use?   Unix-style
       numeric user-id/group-id's?   X.500 Distinguished Names in ACL's?
       Windows-style Security ID's?
   * Do you assume that all of the machines in your distributed
       computation system belong to the same administrative domain?
       What if individuals owning their own workstations want to have
       system administrator privs on their system?  Or is your
       distributed OS a niche system which only works when you have
       clusters of machines that are all centrally and
       administratively owned?
   * What scale should the distributed system work at?  10's of machines
       in a cluster?   100's of machines?  1000's of machines?
       Tens of thousands of machines?  Distributed systems that work
       well on football-sized data centers may not work that well
       when you only have a few racks in colo facility.   The "I forgot
       how to count that low" challenge is a real one....

There have been many, many proposals in the distributed computing
arena which all try to answer these questions differently.  Solaris
had an answer with Yellow Pages, NFS, etc.  OSF/DCE had an answer
involving Kerberos, DCE/RPC, DCE/DFS, etc.  More recently we have
Docker's Swarm and Kubernetes, etc.  None have achieved dominance, and
that should tell us something.

The advantage of trying push all of these questions into the OS is
that you can try to provide the illusion that there is no difference
between local and remote resources.  But that either means that you
have a toy (sorry, "research") system which ignores all of the ways in
which remote computation which extends to a different node that may or
may not be up, which may or may not have belong to a different
administration domain, which may or may not have an adversary on the
network between you and the remote node, etc.  OR, you have to make
access to local resources just as painful as access to remote
resources.  Furthermore, since supporting access remote resources is
going to have more overhead, the illusion that access to local and
remote resources can be the same can't be comfortably sustained in any
case.

When you add to that the complexities of building an OS that tries to
do a really good job supporting local resources --- see all of the
observations in Rob Pike's Systems Software Research is Dead slides
about why this is hard --- it seems to me the solution of trying to
build a hard dividing line between the Local OS and Distributed
Computation infrastructure is the right one.

There is a huge difference between creating a local OS that can live
on a single developer's machine in their house --- and a distributed
OS which requires setting up a directory server, and an authentication
server, and a secure distributed time server, etc., before you set up
the first useful node that can actually run user workloads.  You can
try to do both under a single source tree, but it's going to result in
a huge amount of bloat, and a huge amount of maintenance burden to
keep it all working.

By keeping the local node OS and the distributed computation system
separate, it can help control complexity, and that's a big part of
computer science, isn't it?

						- Ted

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-01 21:58 [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
  2021-09-02  8:42 ` Tony Finch
  2021-09-02 15:41 ` Kevin Bowling
@ 2021-09-03 15:56 ` Warner Losh
  2021-09-03 17:10   ` Adam Thornton
  2021-09-16 18:38 ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
  3 siblings, 1 reply; 52+ messages in thread
From: Warner Losh @ 2021-09-03 15:56 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Wed, Sep 1, 2021 at 4:00 PM Dan Cross <crossd@gmail.com> wrote:

> I'm curious about other peoples' thoughts on the talk and the overall
> topic?
>

My comment is that the mental map that he presents has always been a lie.
At least it's been a lie from a very early time.

Even in Unibus/Qbus days, the add-in cards had some kind of processor
on it from an early time. Several of the VAX boards had 68000 or similar
CPUs that managed memory. Even the simpler MFM boards had buffer
memory that needed to be managed before the DMA/PIO pulled it out
of the card. There's always been an element of different address spaces
with different degrees of visibility into those address spaces.

What has changed is all of these things are now on the SoC die so
you have good visibility (well, as good as the docs) into these things.
The number of different things has increased, and the for cross domain
knowledge has increased.

The simplistic world view was even inaccurate at the start....

Warner

[-- Attachment #2: Type: text/html, Size: 1545 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 15:56 ` Warner Losh
@ 2021-09-03 17:10   ` Adam Thornton
  2021-09-03 17:28     ` Larry McVoy
  2021-09-03 17:46     ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware [ really a comment on SoCs ] Jon Steinhart
  0 siblings, 2 replies; 52+ messages in thread
From: Adam Thornton @ 2021-09-03 17:10 UTC (permalink / raw)
  To: Warner Losh, The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2384 bytes --]

Much of the problem, I think, is that:

1) an idealized PDP-11  (I absolutely take Warner's point that that
idealization never really existed) is a sufficiently simple model that a
Bear Of Little Brain, such as myself, can reason about what's going to
happen in response to a particular sequence of instructions, and get fairly
proficient in instructing the machine to do so in a non-geological
timeframe.

2) a modern CPU?  Let alone SoC?  Fuggedaboutit unless you're way, way
smarter than I am.  (I mean, I do realize that this particular venue has a
lot of those people in it...but, really, those are people with
extraordinary minds.)

There are enough people in the world capable of doing 1 and not 2 that we
can write software that usually mostly kinda works and often gets stuff
done before collapsing in a puddle of nasty-smelling goo.  There aren't
many people at all capable of 2, and as the complexity of systems
increases, that number shrinks.

In short, this ends up being the same argument that comes around every so
often, "why are you people still pretending that the computer is a PDP-11
when it clearly isn't?"  Because, as with the keys and the streetlight,
that's what we have available to us.  Only a grossly oversimplified model
fits into our heads.

Adam

On Fri, Sep 3, 2021 at 8:57 AM Warner Losh <imp@bsdimp.com> wrote:

>
>
> On Wed, Sep 1, 2021 at 4:00 PM Dan Cross <crossd@gmail.com> wrote:
>
>> I'm curious about other peoples' thoughts on the talk and the overall
>> topic?
>>
>
> My comment is that the mental map that he presents has always been a lie.
> At least it's been a lie from a very early time.
>
> Even in Unibus/Qbus days, the add-in cards had some kind of processor
> on it from an early time. Several of the VAX boards had 68000 or similar
> CPUs that managed memory. Even the simpler MFM boards had buffer
> memory that needed to be managed before the DMA/PIO pulled it out
> of the card. There's always been an element of different address spaces
> with different degrees of visibility into those address spaces.
>
> What has changed is all of these things are now on the SoC die so
> you have good visibility (well, as good as the docs) into these things.
> The number of different things has increased, and the for cross domain
> knowledge has increased.
>
> The simplistic world view was even inaccurate at the start....
>
> Warner
>

[-- Attachment #2: Type: text/html, Size: 3365 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 17:10   ` Adam Thornton
@ 2021-09-03 17:28     ` Larry McVoy
  2021-09-03 17:42       ` John Floren
                         ` (2 more replies)
  2021-09-03 17:46     ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware [ really a comment on SoCs ] Jon Steinhart
  1 sibling, 3 replies; 52+ messages in thread
From: Larry McVoy @ 2021-09-03 17:28 UTC (permalink / raw)
  To: Adam Thornton; +Cc: The Eunuchs Hysterical Society

I am exactly as Adam described, still thinking like it is a PDP-11.
Such an understandable machine.   For me, out of order execution kind
of blew up my brain, that's when I stopped doing serious kernel work,
I just couldn't get to a mental model of how you reasoned about that.

Though I was talking to someone about it, maybe Clem, recently and
came to the conclusion that it is fine, we already sort of had this
mess with pipelines.  So maybe it is fine, but out of order bugs my
brain.

On Fri, Sep 03, 2021 at 10:10:57AM -0700, Adam Thornton wrote:
> Much of the problem, I think, is that:
> 
> 1) an idealized PDP-11  (I absolutely take Warner's point that that
> idealization never really existed) is a sufficiently simple model that a
> Bear Of Little Brain, such as myself, can reason about what's going to
> happen in response to a particular sequence of instructions, and get fairly
> proficient in instructing the machine to do so in a non-geological
> timeframe.
> 
> 2) a modern CPU?  Let alone SoC?  Fuggedaboutit unless you're way, way
> smarter than I am.  (I mean, I do realize that this particular venue has a
> lot of those people in it...but, really, those are people with
> extraordinary minds.)
> 
> There are enough people in the world capable of doing 1 and not 2 that we
> can write software that usually mostly kinda works and often gets stuff
> done before collapsing in a puddle of nasty-smelling goo.  There aren't
> many people at all capable of 2, and as the complexity of systems
> increases, that number shrinks.
> 
> In short, this ends up being the same argument that comes around every so
> often, "why are you people still pretending that the computer is a PDP-11
> when it clearly isn't?"  Because, as with the keys and the streetlight,
> that's what we have available to us.  Only a grossly oversimplified model
> fits into our heads.
> 
> Adam
> 
> On Fri, Sep 3, 2021 at 8:57 AM Warner Losh <imp@bsdimp.com> wrote:
> 
> >
> >
> > On Wed, Sep 1, 2021 at 4:00 PM Dan Cross <crossd@gmail.com> wrote:
> >
> >> I'm curious about other peoples' thoughts on the talk and the overall
> >> topic?
> >>
> >
> > My comment is that the mental map that he presents has always been a lie.
> > At least it's been a lie from a very early time.
> >
> > Even in Unibus/Qbus days, the add-in cards had some kind of processor
> > on it from an early time. Several of the VAX boards had 68000 or similar
> > CPUs that managed memory. Even the simpler MFM boards had buffer
> > memory that needed to be managed before the DMA/PIO pulled it out
> > of the card. There's always been an element of different address spaces
> > with different degrees of visibility into those address spaces.
> >
> > What has changed is all of these things are now on the SoC die so
> > you have good visibility (well, as good as the docs) into these things.
> > The number of different things has increased, and the for cross domain
> > knowledge has increased.
> >
> > The simplistic world view was even inaccurate at the start....
> >
> > Warner
> >

-- 
---
Larry McVoy            	     lm at mcvoy.com             http://www.mcvoy.com/lm 

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 17:28     ` Larry McVoy
@ 2021-09-03 17:42       ` John Floren
  2021-09-03 19:02       ` Lawrence Stewart
  2021-09-03 19:11       ` Clem Cole
  2 siblings, 0 replies; 52+ messages in thread
From: John Floren @ 2021-09-03 17:42 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

When I took Computer Architecture, "reasoning" about out-of-order execution involved 30-page worksheets where we could track the state of the Tomasulo algorithm through each iteration. It was ludicrously slow work, and wouldn't be a lot of fun even if you had a computerized tool to help step through things instead.

If you're talking about a modern Intel CPU where your compiler emits CISC instructions which are actually implemented in RISC instructions in the microcode, which in turn get rewritten and reordered internally by the CPU... it's hard to fault programmers for thinking at the level of the instruction set that's presented to them, even if it looks like a PDP-11.

The above should not be read as an endorsement of the CPU status quo, of course :)

john

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

On Friday, September 3rd, 2021 at 10:28 AM, Larry McVoy <lm@mcvoy.com> wrote:

> I am exactly as Adam described, still thinking like it is a PDP-11.
>
> Such an understandable machine. For me, out of order execution kind
>
> of blew up my brain, that's when I stopped doing serious kernel work,
>
> I just couldn't get to a mental model of how you reasoned about that.
>
> Though I was talking to someone about it, maybe Clem, recently and
>
> came to the conclusion that it is fine, we already sort of had this
>
> mess with pipelines. So maybe it is fine, but out of order bugs my
>
> brain.
>
> On Fri, Sep 03, 2021 at 10:10:57AM -0700, Adam Thornton wrote:
>
> > Much of the problem, I think, is that:
> >
> > 1.  an idealized PDP-11 (I absolutely take Warner's point that that
> >
> >     idealization never really existed) is a sufficiently simple model that a
> >
> >     Bear Of Little Brain, such as myself, can reason about what's going to
> >
> >     happen in response to a particular sequence of instructions, and get fairly
> >
> >     proficient in instructing the machine to do so in a non-geological
> >
> >     timeframe.
> >
> > 2.  a modern CPU? Let alone SoC? Fuggedaboutit unless you're way, way
> >
> >     smarter than I am. (I mean, I do realize that this particular venue has a
> >
> >     lot of those people in it...but, really, those are people with
> >
> >     extraordinary minds.)
> >
> >
> > There are enough people in the world capable of doing 1 and not 2 that we
> >
> > can write software that usually mostly kinda works and often gets stuff
> >
> > done before collapsing in a puddle of nasty-smelling goo. There aren't
> >
> > many people at all capable of 2, and as the complexity of systems
> >
> > increases, that number shrinks.
> >
> > In short, this ends up being the same argument that comes around every so
> >
> > often, "why are you people still pretending that the computer is a PDP-11
> >
> > when it clearly isn't?" Because, as with the keys and the streetlight,
> >
> > that's what we have available to us. Only a grossly oversimplified model
> >
> > fits into our heads.
> >
> > Adam
> >
> > On Fri, Sep 3, 2021 at 8:57 AM Warner Losh imp@bsdimp.com wrote:
> >
> > > On Wed, Sep 1, 2021 at 4:00 PM Dan Cross crossd@gmail.com wrote:
> > >
> > > > I'm curious about other peoples' thoughts on the talk and the overall
> > > >
> > > > topic?
> > >
> > > My comment is that the mental map that he presents has always been a lie.
> > >
> > > At least it's been a lie from a very early time.
> > >
> > > Even in Unibus/Qbus days, the add-in cards had some kind of processor
> > >
> > > on it from an early time. Several of the VAX boards had 68000 or similar
> > >
> > > CPUs that managed memory. Even the simpler MFM boards had buffer
> > >
> > > memory that needed to be managed before the DMA/PIO pulled it out
> > >
> > > of the card. There's always been an element of different address spaces
> > >
> > > with different degrees of visibility into those address spaces.
> > >
> > > What has changed is all of these things are now on the SoC die so
> > >
> > > you have good visibility (well, as good as the docs) into these things.
> > >
> > > The number of different things has increased, and the for cross domain
> > >
> > > knowledge has increased.
> > >
> > > The simplistic world view was even inaccurate at the start....
> > >
> > > Warner
>
> Larry McVoy lm at mcvoy.com http://www.mcvoy.com/lm

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware [ really a comment on SoCs ]
  2021-09-03 17:10   ` Adam Thornton
  2021-09-03 17:28     ` Larry McVoy
@ 2021-09-03 17:46     ` Jon Steinhart
  1 sibling, 0 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-03 17:46 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

Adam Thornton writes:
>
> 2) a modern CPU?  Let alone SoC?  Fuggedaboutit unless you're way, way

Don't know how many of you are hardware/software folks as opposed to just
software, but it turns out that SoCs are not immune to the problems that
result from bolting disparate stuff together in the software world.

A few years ago I was working on a project that included an Atmel SoC.
Was having some weirdness, and contacted Atmel to ask them about timing
issues between two of the DMA controllers that was unspecified in their
documentation but critical to making the project work.  Their initial
response was "Well, we don't know, that's IP that we bought from ARM,
call them and ask."  I replied "I don't think that they're gonna talk to
me because I'm not the one that bought that IP; you did and it's your job."
Fortunately, at that time I knew someone who could escalate the issue for
me, and only two months later got the timing numbers that I needed.

One could be understandably think that more attention is given to bolting
IP blocks together on an SoC than is given when bolting a thousand libraries
onto a simple program but that turns out to not be true.  Software folks
need to be prepared for the fact that there may actually be nobody who
knows how portions of the hardware actually work because it's not all
designed at one place anymore.  We have to be prepared for the SoC version
of node.js.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 17:28     ` Larry McVoy
  2021-09-03 17:42       ` John Floren
@ 2021-09-03 19:02       ` Lawrence Stewart
  2021-09-03 19:11       ` Clem Cole
  2 siblings, 0 replies; 52+ messages in thread
From: Lawrence Stewart @ 2021-09-03 19:02 UTC (permalink / raw)
  To: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1747 bytes --]

I don’t really think out-of-order in hardware causes trouble for programmers that wasn’t already there when you use -O3.  Compilers will already promote memory to registers and do interprocedural optimization and reorder memory references.  You have to sprinkle 
asm volatile("" ::: "memory");
around like pixie dust to make sure the compiler does things in the order you want, nevermind the hardware.

x86 has wildly complex microarchitecture, but the model for a single thread is perfectly sensible.  It seems to work like it was in-order. OOO isn’t changing that model of execution at all.  I mean you care about it when performance tuning, but not for correctness.

Other architectures, ARM, IBM, Alpha, are far worse in this respect.

The real problems are when you have multicore in a shared memory space and you have to read all the fine print in the memory ordering chapter and understand the subtle differences between LFENCE and SFENCE and MFENCE.  I get that, but I also think shared memory is a failed experiment and we should have gone with distributed memory and clusters and message passing.  It is possible for mortals to program those.    Regular folks can make 1000 rank MPI programs work, where almost noone other than Maurice Herlihy or like that can reason about locks and threads and shared data structures.

My wish for a better world is to integrate messaging into the architecture rather than have an I/O device model for communications.  It is criminal that machine to machine comms are still stuck at 800 nanoseconds or so latency.  It takes 200 instructions or so to send a message under the best circumstances and a similar number to receive it, plus bus, adapter, wire, and switch time. 

-L




[-- Attachment #2: Type: text/html, Size: 8199 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 17:28     ` Larry McVoy
  2021-09-03 17:42       ` John Floren
  2021-09-03 19:02       ` Lawrence Stewart
@ 2021-09-03 19:11       ` Clem Cole
  2 siblings, 0 replies; 52+ messages in thread
From: Clem Cole @ 2021-09-03 19:11 UTC (permalink / raw)
  To: Larry McVoy; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 6351 bytes --]

Larry - it's the compiler (code generator) folks that I really feel bad
for. They had to deal with the realities of the ISA in many ways more than
we do and for them, it's getting worse and worse.  BTW - there was a
misstatement in a previous message. A current CISC system like the INTEL*64
is not implemented as a RISC µcode, nor are current more RISCy machines
like the SPARX and Alpha much less the StrongARM and its followers.   What
they are internally are *data flow machines *which is why you getting a
mixing of instruction ordering, scoreboarding, and all sorts of
complexities that blows our mind.

 At least at the OS we have been used to doing things in parallel,
exceptions and interrupts occurring and we have reasoned our ways through
things.  Butler Lampson and Leslie Lamport gave a parallel calculus to help
verify things (although Butler once observed at an old SOSP talk that the
problem with parallel is what does 'single step the processor mean
anymore.' ).

So the idea while the processor is not a PDP-10 or PDP-11 much less a
360/91 or a CDC-6600, we build a model in our heads that does simplify the
machine(s) as much as possible.  We ensure at least that is correct and
then, build up more complexity from there.

To me, the problem is that we too often do a poor job of what should be the
simple stuff and we continue to make it too complicated.  Not to pick on
any one group/code base, but Jon's recent observation about the Linux
kernel FS interface is a prime point.  It's not the processor that was made
complex, it's the SW wanting to be all things to all people.

To me what Unix started and succeed at its time, and clearly Plan9 was
attempted in its time (but failed commercially) was to mask if not toss out
as much of the complexity of the HW and get to a couple of simple and
common ideas and all programs could agree.  Going back to the idea of the
bear of the 'slittle brain and try to expose the simplest way to
computation.

Two of the best Unix talks/papers ever, Rob's "cat -v is a bad idea" and
Tom's "All Chips that Fit" has morphed into "I have 64-bits of address
space I can link anything into my framework" and  "what I power and cool in
my current process technology" [a SoC is not different that the board level
products that some of us lived].

I recently read a suggestion that the best way to teach begging students to
be "good programmers" was to "introduce them to as many frameworks as
possible and teach as little theory as they need." I nearly lost my
dinner.  Is this what programming has come to?
 Framework/Access Methods/Smart Objects ....   To be fair, my own employer
is betting on DPC++ and believing OneAPI as the one ring to rule them all.

There is a lot to be said of "small is beautiful."  How did we get from
Sixth Edition UNIX with K&R1 to today?  One transistor and one line a code
at a time.

ᐧ

On Fri, Sep 3, 2021 at 1:29 PM Larry McVoy <lm@mcvoy.com> wrote:

> I am exactly as Adam described, still thinking like it is a PDP-11.
> Such an understandable machine.   For me, out of order execution kind
> of blew up my brain, that's when I stopped doing serious kernel work,
> I just couldn't get to a mental model of how you reasoned about that.
>
> Though I was talking to someone about it, maybe Clem, recently and
> came to the conclusion that it is fine, we already sort of had this
> mess with pipelines.  So maybe it is fine, but out of order bugs my
> brain.
>
> On Fri, Sep 03, 2021 at 10:10:57AM -0700, Adam Thornton wrote:
> > Much of the problem, I think, is that:
> >
> > 1) an idealized PDP-11  (I absolutely take Warner's point that that
> > idealization never really existed) is a sufficiently simple model that a
> > Bear Of Little Brain, such as myself, can reason about what's going to
> > happen in response to a particular sequence of instructions, and get
> fairly
> > proficient in instructing the machine to do so in a non-geological
> > timeframe.
> >
> > 2) a modern CPU?  Let alone SoC?  Fuggedaboutit unless you're way, way
> > smarter than I am.  (I mean, I do realize that this particular venue has
> a
> > lot of those people in it...but, really, those are people with
> > extraordinary minds.)
> >
> > There are enough people in the world capable of doing 1 and not 2 that we
> > can write software that usually mostly kinda works and often gets stuff
> > done before collapsing in a puddle of nasty-smelling goo.  There aren't
> > many people at all capable of 2, and as the complexity of systems
> > increases, that number shrinks.
> >
> > In short, this ends up being the same argument that comes around every so
> > often, "why are you people still pretending that the computer is a PDP-11
> > when it clearly isn't?"  Because, as with the keys and the streetlight,
> > that's what we have available to us.  Only a grossly oversimplified model
> > fits into our heads.
> >
> > Adam
> >
> > On Fri, Sep 3, 2021 at 8:57 AM Warner Losh <imp@bsdimp.com> wrote:
> >
> > >
> > >
> > > On Wed, Sep 1, 2021 at 4:00 PM Dan Cross <crossd@gmail.com> wrote:
> > >
> > >> I'm curious about other peoples' thoughts on the talk and the overall
> > >> topic?
> > >>
> > >
> > > My comment is that the mental map that he presents has always been a
> lie.
> > > At least it's been a lie from a very early time.
> > >
> > > Even in Unibus/Qbus days, the add-in cards had some kind of processor
> > > on it from an early time. Several of the VAX boards had 68000 or
> similar
> > > CPUs that managed memory. Even the simpler MFM boards had buffer
> > > memory that needed to be managed before the DMA/PIO pulled it out
> > > of the card. There's always been an element of different address spaces
> > > with different degrees of visibility into those address spaces.
> > >
> > > What has changed is all of these things are now on the SoC die so
> > > you have good visibility (well, as good as the docs) into these things.
> > > The number of different things has increased, and the for cross domain
> > > knowledge has increased.
> > >
> > > The simplistic world view was even inaccurate at the start....
> > >
> > > Warner
> > >
>
> --
> ---
> Larry McVoy                  lm at mcvoy.com
> http://www.mcvoy.com/lm
>

[-- Attachment #2: Type: text/html, Size: 9075 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 13:21       ` Theodore Ts'o
@ 2021-09-08 11:14         ` Tony Finch
  2021-09-16 19:27         ` Dan Cross
  1 sibling, 0 replies; 52+ messages in thread
From: Tony Finch @ 2021-09-08 11:14 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list

Theodore Ts'o <tytso@mit.edu> wrote:
>
> There have been many, many proposals in the distributed computing
> arena which all try to answer these questions differently.  Solaris
> had an answer with Yellow Pages, NFS, etc.  OSF/DCE had an answer
> involving Kerberos, DCE/RPC, DCE/DFS, etc.  More recently we have
> Docker's Swarm and Kubernetes, etc.  None have achieved dominance, and
> that should tell us something.

I think there are two different kinds of distributed computing there.

Distributed authentication and administration is dominated by Microsoft
Active Directory (LDAP, Kerberos, DNS, SMB, ...) which I think can
reasonably be regarded as part of Windows (even if many Windows machines
aren't part of an AD). That kind of distributed system doesn't try to help
you stop caring that there are lots of computers.

Whereas Kubernetes and Docker Swarm do automatic lifecycle management for
distributed workloads. I have not yet had the pleasure (?) of working with
them but I get the impression that it's difficult to set up their access
control to stop giving everything root on everything else. They try much
harder to make a cluster work as a single system.

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  https://dotat.at/
St Davids Head to Great Orme Head, including St Georges Channel:
Easterly or southeasterly 2 to 4, occasionally 5 at first in north,
becoming variable 2 to 4 later. Smooth or slight. Showers, perhaps
thundery, fog patches later. Moderate or good, occasionally very poor
later.


^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-01 21:58 [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
                   ` (2 preceding siblings ...)
  2021-09-03 15:56 ` Warner Losh
@ 2021-09-16 18:38 ` Dan Cross
  2021-09-16 19:34   ` Jon Steinhart
  3 siblings, 1 reply; 52+ messages in thread
From: Dan Cross @ 2021-09-16 18:38 UTC (permalink / raw)
  To: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 2748 bytes --]

On Wed, Sep 1, 2021 at 5:58 PM Dan Cross <crossd@gmail.com> wrote:

> [snip]
>

First, thank you for all of the thoughtful responses, both on-list and off.

An interesting theme in many of the responses was essentially questioning
whether the underlying OS still matters, since the focus on development has
shifted to higher levels? E.g., we now provision components of our
enormously large and complicated distributed applications with
building blocks like containers, less physical machines, let alone
processes etc. That is certainly a trend, but it strikes me that those
containers have to run somewhere, and at some point, we've still got
instructions executing on some CPU, modifying words of memory, registers,
etc; presumably all of that runs under the control of an operating system.

It is a worthwhile question to ask whether that operating system still
matters at all: what we have works, and since it's so hidden behind layers
upon layers of abstraction, do we really care what it is? But I claim that
it does perhaps more than most folks realize. Certainly, there are metrics
that people care about (tail latency, jitter, efficiency at the 90th, 95th,
99th percentile...) and OS effects can have outsized impacts there; Mothy's
talk alludes to this when he talks about all of the hidden processing
that's happening all over a modern computer, eventually some of that
trickles onto the cores that are running one's containerized Node
application or whatever (lookin' at you, SMM mode...). At the end of the
day, the code we care about still runs in some process under some OS on
some bit of physical hardware, regardless of all of the abstractions we've
placed on top of those things. What that system does, and the abstractions
that its interface provides to programs, still matters.

Perhaps another question worth asking is, does it make sense to look at
different models for those systems? My subjective impression is that, back
in the 60s and 70s, there was much greater variation in system
architectures than today. A common explanation for this is that we didn't
know how to build systems at the time, so folks threw a lot of stuff at the
wall to see what would stick. But we no longer do that...again, Mothy
alludes to this in his brief survey of OSDI papers: basically, new systems
aren't being presented. Rob Pike also lamented that state of affairs 20
years ago, so it's been going on for a while. Does that mean that we've
come up with a recipe for systems that work and work well, and therefore we
don't need to rethink those basic building blocks? Or does that mean that
we're so used to our systems working well enough that we've become myopic
about their architecture, and thus blind to their faults?

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 3287 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-03 13:21       ` Theodore Ts'o
  2021-09-08 11:14         ` Tony Finch
@ 2021-09-16 19:27         ` Dan Cross
  2021-09-17  0:34           ` Theodore Ts'o
  1 sibling, 1 reply; 52+ messages in thread
From: Dan Cross @ 2021-09-16 19:27 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Douglas McIlroy

[-- Attachment #1: Type: text/plain, Size: 10908 bytes --]

On Fri, Sep 3, 2021 at 9:23 AM Theodore Ts'o <tytso@mit.edu> wrote:

> On Thu, Sep 02, 2021 at 11:24:37PM -0400, Douglas McIlroy wrote:
> > I set out to write a reply, then found that Marshall had said it all,
> > better..Alas, the crucial central principle of Plan 9 got ignored, while
> > its ancillary contributions were absorbed into Linux, making Linux fatter
> > but still oriented to a bygone milieu.
>
> I'm really not convinced trying to build distributed computing into
> the OS ala Plan 9 is viable.


It seems like plan9 itself is an existence proof that this is possible.
What it did not present was an existence proof of its scalability and it
wasn't successful commercially. It probably bears mentioning that that
wasn't really the point of plan9, though; it was a research system.

I'll try to address the plan9 specific bits below.

The moment the OS has to span multiple
> TCB's (Trusted Computing Bases), you have to make some very
> opinionated decisions on a number of issues for which we do not have
> consensus after decades of trial and error:
>

Interestingly, plan9 did make opinionated decisions about all of the things
mentioned below. Largely, those decisions worked out pretty well.

   * What kind of directory service do you use?  x.500/LDAP?   Yellow Pages?
>        Project Athena's Hesiod?
>

In the plan9 world, the directory services were provided by the filesystem
and a user-level library that consumed databases of information resident in
the filesystem itself (ndb(6) -- https://9p.io/magic/man2html/6/ndb). It
also provided DNS services for interacting with the larger world. The
connection server was an idea that was ahead of it's time (ndb(8) --
https://9p.io/magic/man2html/8/ndb). Also,
https://9p.io/sys/doc/net/net.html.

   * What kind of distributed authentication do you use?  Kerboers?
>        Trust on first use authentication ala ssh?  .rhosts style
>        "trust the network" style authentication?
>

Plan 9 specifically used a Kerberos-like model, but did not use the
Kerberos protocol. I say "Kerberos-like" in that there was a trusted agent
on the network that provided authentication services using a protocol based
on shared secrets.

   * What kind of distributed authorization service do you use?   Unix-style
>        numeric user-id/group-id's?   X.500 Distinguished Names in ACL's?
>        Windows-style Security ID's?
>

User and group names were simple strings. There were no numeric UIDs
associated with processes, though the original file server had a
user<->integer mapping for directory entries.

   * Do you assume that all of the machines in your distributed
>        computation system belong to the same administrative domain?
>

Not necessarily, no. The model is one of resource sharing, rather than
remote access. You usually pull resources into your local namespace, and
those can come from anywhere you have access to take them from. They may
come from a different "administrative domain". For example, towards the end
of the BTL reign, there was a file server at Bell Labs that some folks had
accounts on and that one could "import" into one's namespace. That was the
main distribution point for the larger community.

       What if individuals owning their own workstations want to have
>        system administrator privs on their system?


When a user logs into a Plan 9 terminal, they become the "host owner" of
that terminal. The host owner is distinguished only in owning the hardware
resources of the hosting machine; they have no other special privileges,
nor is there an administrative user like `root`. It's slightly unusual,
though not unheard of, for a terminal to have a local disk; the disk device
is implicitly owned by the host owner. If the user puts a filesystem on
that device (say, to cache a dataset locally or something), that's on them,
though host owner status doesn't really give any special privileges over
the permissions on the files on that filesystem, modulo them going through
the raw disk device, of course. That is, the uid==0 implies you bypass all
access permission checking is gone in Plan 9. CPU servers have a mechanism
where a remote user can start a process on the server that becomes owned by
the calling user; this is similar to remote login, except that the user
brings their environment with them; the model is more of importing the CPU
server's computational resources into the local environment than, say,
ssh'ing into a machine.

Or is your
>        distributed OS a niche system which only works when you have
>        clusters of machines that are all centrally and
>        administratively owned?
>

I'm not sure how to parse that; I mean, arguably networks of Unix machines
associated with any given organization are "centrally owned and
administered"? To get access to any given plan9 network, someone would have
to create an account on the file and auth servers, but the system was also
installable on a standalone machine with a local filesystem. If folks
wanted to connect in from other types of systems, there were mechanisms for
doing so: `ssh` and `telnet` servers were distributed and could be used,
though the experience for an interactive user was pretty anemic. It was
more typical to use a program called `drawterm` that runs as an application
on e.g. a Mac or Unix machine and emulates enough of a Plan 9 terminal
kernel that a user can effectively `cpu` to a plan9 CPU server. Once logged
in via drawterm, one can run an environment including a window system and
all the graphical stuff from there.

Perhaps the aforementioned Bell Labs file server example clarifies things a
bit?

   * What scale should the distributed system work at?  10's of machines
>        in a cluster?   100's of machines?  1000's of machines?
>        Tens of thousands of machines?


This is, I think, where one gets to the crux of the problem. Plan 9 worked
_really_ well for small clusters of machines (10s) and well enough for
larger clusters (up to 100s or 1000s).

Distributed systems that work
>        well on football-sized data centers may not work that well
>        when you only have a few racks in colo facility.  The "I forgot
>        how to count that low" challenge is a real one...


And how. Plan 9 _was_ eventually ported to football-field sized machines
(the BlueGene port for DoE was on that scale); Ron can be able to speak to
that in more detail, if he is so inclined. In fairness, I do think that
required significant effort and it was, of course, highly specialized to
HPC applications.

My subjective impression was that any given plan9 network would break down
at the scale of single-digit thousands of machines and, perhaps, tens of
thousands of users. Growing beyond that for general use would probably
require some pretty fundamental changes; for example, 9P (the file
protocol) includes a client-chosen "FID" in transactions related to any
open file, so that file servers must keep track of client state to
associate fids to actual files, whether those file refer to disk-resident
stable storage or software synthesized "files" for other things (IPC end
points; process memory; whatever).

There have been many, many proposals in the distributed computing
> arena which all try to answer these questions differently.  Solaris
> had an answer with Yellow Pages, NFS, etc.  OSF/DCE had an answer
> involving Kerberos, DCE/RPC, DCE/DFS, etc.  More recently we have
> Docker's Swarm and Kubernetes, etc.  None have achieved dominance, and
> that should tell us something.
>
> The advantage of trying push all of these questions into the OS is
> that you can try to provide the illusion that there is no difference
> between local and remote resources.


Is that the case, or is it that system designers try to provide uniform
access to different classes of resources? Unix treats socket descriptors
very similar to file descriptors very similar to pipes; why shouldn't named
resources be handled in similar ways?

But that either means that you
> have a toy (sorry, "research") system which ignores all of the ways in
> which remote computation which extends to a different node that may or
> may not be up, which may or may not have belong to a different
> administration domain, which may or may not have an adversary on the
> network between you and the remote node, etc.  OR, you have to make
> access to local resources just as painful as access to remote
> resources.  Furthermore, since supporting access remote resources is
> going to have more overhead, the illusion that access to local and
> remote resources can be the same can't be comfortably sustained in any
> case.
>

...or some other way, which we'll never know about because no one thinks to
ask the question, "how could we do this differently?" I think that's the
crux of Mothy's point.

Plan 9, as just one example, asked a lot of questions about the issues you
mentioned above 30 years ago. They came up with _a_ set of answers; that
set did evolve over time as things progressed. That doesn't mean that those
questions were resolved definitively, just that there was a group of
researchers who came up with an approach to them that worked for that group.

What's changed is that we now take for granted that Linux is there, and
we've stopped asking questions about anything outside of that model.

When you add to that the complexities of building an OS that tries to
> do a really good job supporting local resources --- see all of the
> observations in Rob Pike's Systems Software Research is Dead slides
> about why this is hard --- it seems to me the solution of trying to
> build a hard dividing line between the Local OS and Distributed
> Computation infrastructure is the right one.
>
> There is a huge difference between creating a local OS that can live
> on a single developer's machine in their house --- and a distributed
> OS which requires setting up a directory server, and an authentication
> server, and a secure distributed time server, etc., before you set up
> the first useful node that can actually run user workloads.  You can
> try to do both under a single source tree, but it's going to result in
> a huge amount of bloat, and a huge amount of maintenance burden to
> keep it all working.


I agree with the first part of this paragraph, but then we're talking about
researchers, not necessarily unaffiliated open-source developers. Hopefully
researchers have some organizational and infrastructure support!

By keeping the local node OS and the distributed computation system
> separate, it can help control complexity, and that's a big part of
> computer science, isn't it?
>

I don't know. It seems like this whole idea of distributed systems built on
networks of loosely coupled, miniature timesharing systems has led to
enormous amounts of inescapable complexity.

I'll bet the Kubernetes by itself is larger than all of plan9.

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 14379 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 18:38 ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
@ 2021-09-16 19:34   ` Jon Steinhart
  2021-09-16 19:41     ` Larry McVoy
  2021-09-17 15:56     ` Bakul Shah
  0 siblings, 2 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-16 19:34 UTC (permalink / raw)
  To: TUHS main list

Dan Cross writes:
>
> First, thank you for all of the thoughtful responses, both on-list and off.
>
> An interesting theme in many of the responses was essentially questioning
> whether the underlying OS still matters, since the focus on development has
> shifted to higher levels? E.g., we now provision components of our
> enormously large and complicated distributed applications with
> building blocks like containers, less physical machines, let alone
> processes etc. That is certainly a trend, but it strikes me that those
> containers have to run somewhere, and at some point, we've still got
> instructions executing on some CPU, modifying words of memory, registers,
> etc; presumably all of that runs under the control of an operating system.
>
> It is a worthwhile question to ask whether that operating system still
> matters at all: what we have works, and since it's so hidden behind layers
> upon layers of abstraction, do we really care what it is? But I claim that
> it does perhaps more than most folks realize. Certainly, there are metrics
> that people care about (tail latency, jitter, efficiency at the 90th, 95th,
> 99th percentile...) and OS effects can have outsized impacts there; Mothy's
> talk alludes to this when he talks about all of the hidden processing
> that's happening all over a modern computer, eventually some of that
> trickles onto the cores that are running one's containerized Node
> application or whatever (lookin' at you, SMM mode...). At the end of the
> day, the code we care about still runs in some process under some OS on
> some bit of physical hardware, regardless of all of the abstractions we've
> placed on top of those things. What that system does, and the abstractions
> that its interface provides to programs, still matters.
>
> Perhaps another question worth asking is, does it make sense to look at
> different models for those systems? My subjective impression is that, back
> in the 60s and 70s, there was much greater variation in system
> architectures than today. A common explanation for this is that we didn't
> know how to build systems at the time, so folks threw a lot of stuff at the
> wall to see what would stick. But we no longer do that...again, Mothy
> alludes to this in his brief survey of OSDI papers: basically, new systems
> aren't being presented. Rob Pike also lamented that state of affairs 20
> years ago, so it's been going on for a while. Does that mean that we've
> come up with a recipe for systems that work and work well, and therefore we
> don't need to rethink those basic building blocks? Or does that mean that
> we're so used to our systems working well enough that we've become myopic
> about their architecture, and thus blind to their faults?
>
>         - Dan C.

Well, I have two different thoughts on this question.

The obvious one to me is that of course it matters.

I'll start by drawing a lame analogy as our 12 week deck reconstruction
project just finished up an hour ago.  In a conversation with the
contractor he said "You're technically doing a repair so you don't have
to pull a permit and bring it up to current code.  But, there are some
things that I recommend that you do anyway because the technology is much
better than what it was when this thing was built 30 years ago."  My point
here is that sure, we can survive with what's there, but that doesn't mean
that we should ignore new technology.

I had dinner with one of the people I mentor a few weeks ago.  He told me
that while he was generally ok working for Google and making gobs of money
that he felt sort of empty.  He told me that if his project got cancelled
he didn't think that the world would notice, and as a mid-20s person seeing
the planet that he's living on burning up, he's thinking about changing
careers to something that's more meaningful (to him).  I think that there is
still plenty of work to do on things like efficiency that would be easy to
justify if politically energy wasn't so heavily subsidized.  Think back a
few decades to all of the power-saving changes that the OLPC spawned.  Most
felt like things were fine the way they were back then too.  I think that
layers upon layers is wasteful.

As I've said before, I'm having difficulty distinguishing the "full stack"
in full stack programming from a compost heap.  It's not OK to me from a
security, safety, and reliability perspective to build on a rotting
foundation.

It's my opinion that the whole container thing sort of started as a "we
can't secure the underlying system so we'll build something secure on top"
combined with "it's no fun to fix the unnecessary incompatible mess among
virtually identical systems that we've made so we'll build a new fix-it
layer" ideologies.  How long until problems are found with containers
it's decided that the way to fix it is to build "safe deposit boxes" that
run in container?  Is there ever an end in sight?

My second train of thought is to ask the question "what is a computer?"
Having started in an era where computing meant basking in the warm glow
of vacuum tubes, it's easy to think that it's the hardware.  But somewhere
along the line microcode made an appearance, followed by nanocode and so
on.  I'm guessing, especially from spending time with current CS students,
that to many a computer is some high-level abstraction.  So the question is,
below what layer is considered a computer today?

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 19:34   ` Jon Steinhart
@ 2021-09-16 19:41     ` Larry McVoy
  2021-09-16 23:14       ` Marshall Conover
                         ` (2 more replies)
  2021-09-17 15:56     ` Bakul Shah
  1 sibling, 3 replies; 52+ messages in thread
From: Larry McVoy @ 2021-09-16 19:41 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Thu, Sep 16, 2021 at 12:34:15PM -0700, Jon Steinhart wrote:
> As I've said before, I'm having difficulty distinguishing the "full stack"
> in full stack programming from a compost heap.  It's not OK to me from a
> security, safety, and reliability perspective to build on a rotting
> foundation.

Amen.

> It's my opinion that the whole container thing sort of started as a "we
> can't secure the underlying system so we'll build something secure on top"
> combined with "it's no fun to fix the unnecessary incompatible mess among
> virtually identical systems that we've made so we'll build a new fix-it
> layer" ideologies.  How long until problems are found with containers
> it's decided that the way to fix it is to build "safe deposit boxes" that
> run in container?  Is there ever an end in sight?

I think it is that the newer kids are less willing to understand stuff.
So they build something on top that they understand.  I agree that they
will hit problems and likely build "safe deposit boxes" because the
containers are "too complex".

Oh, and get off my lawn!

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 19:41     ` Larry McVoy
@ 2021-09-16 23:14       ` Marshall Conover
  2021-09-16 23:44         ` Rob Pike
                           ` (2 more replies)
  2021-09-16 23:45       ` Jon Steinhart
  2021-09-16 23:54       ` David Arnold
  2 siblings, 3 replies; 52+ messages in thread
From: Marshall Conover @ 2021-09-16 23:14 UTC (permalink / raw)
  To: Larry McVoy; +Cc: TUHS main list

While I got a chuckle from this, it focuses on security, and I don't
think security sold docker containers. I think what really sold
containers was their ability to solve the old, hard problems of
configuring and maintaining servers.

Docker's use of per-process namespaces is a godsend for running
different services on the same machine. I no longer run into two
applications fighting over dependency versions, because both
applications are running in their own world. This was somewhat
possible in chroots, but as someone who tried to use chroots that way
a decade ago, docker's made it trivial.

Containers are also a godsend for applications that have to be
deployed somewhere else. I know a container I deploy will have
everything it needs wherever it goes, and will be exactly the thing I
built and tested. It's hard to understate the benefits of this: when
deploying, I no longer run into issues like "oh shoot, there was some
configuration I forgot about on the dev server that I need for prod."
I no longer have to create server configuration documentation either,
as the documentation is "docker build," followed by "docker run." When
we were first starting out on our current project, we built a
container that runs our build system's agents. At one point the VM on
which we were running those agents went down, and our stop-gap fix was
to download and run a few copies of that container locally. As a
result, we had builds going the entire time we worked to fix the
issue.

---------------

Separately, for the larger discussion, I think the
abstraction-aerospace-engineering seen over the last few decades comes
from the adage "necessity is the mother of invention." People writing
business logic today are targeting an OS-independent platform: the
browser. That's where developers need solutions, and that's where we
see movement. Considering this, it's no surprise the browser has
stumbled backwards from a markup language-renderer into a full
platform for downloading and running applications and managing their
resources, as well as providing complex abstractions for interacting
with distributed systems. And it's no surprise those distributed
systems have separated as much as possible from whatever's not the
browser.

In fact, we're seeing agreement in the browser ecosystem for problems
like the directory system choice mentioned above. The OIDC workflow
was born out of the internet's many-users-to-many-services issue. Now,
it's such a decided approach for managing users' access to services
that big names like Amazon and Google offer identity provider services
using it, and I, as a service writer, can swap between any of them
transparently. The services I run only care that the token they're
handed is signed by the auth server they're configured to use, and
that the token says the user is allowed to use the service contacted.
The applications I write and use have no clue what the OS' permissions
are for anything they deal with. For them, OS permissions have been
made redundant.

With this context, I think most of us here have learned by experience
why the OS gets no more development, in every discussion they've had
with management where they've said "we need to refactor some code that
is wonky, but mostly works, because there will probably be errors and
bugs and security issues in the future if we don't." Management -
which in this case, means the world at large - demands new features,
not unspecified heisen-benefits from redoing things that already work.
For new features, the browser is their only recourse.

And, to boot - if you change the thing under the browser, what if it
breaks the browser?

Cheers!

Marshall










On Thu, Sep 16, 2021 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
>
> On Thu, Sep 16, 2021 at 12:34:15PM -0700, Jon Steinhart wrote:
> > As I've said before, I'm having difficulty distinguishing the "full stack"
> > in full stack programming from a compost heap.  It's not OK to me from a
> > security, safety, and reliability perspective to build on a rotting
> > foundation.
>
> Amen.
>
> > It's my opinion that the whole container thing sort of started as a "we
> > can't secure the underlying system so we'll build something secure on top"
> > combined with "it's no fun to fix the unnecessary incompatible mess among
> > virtually identical systems that we've made so we'll build a new fix-it
> > layer" ideologies.  How long until problems are found with containers
> > it's decided that the way to fix it is to build "safe deposit boxes" that
> > run in container?  Is there ever an end in sight?
>
> I think it is that the newer kids are less willing to understand stuff.
> So they build something on top that they understand.  I agree that they
> will hit problems and likely build "safe deposit boxes" because the
> containers are "too complex".
>
> Oh, and get off my lawn!

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:14       ` Marshall Conover
@ 2021-09-16 23:44         ` Rob Pike
  2021-09-17  0:37           ` Larry McVoy
  2021-09-17  1:38         ` Jon Steinhart
  2021-09-17  3:54         ` John Cowan
  2 siblings, 1 reply; 52+ messages in thread
From: Rob Pike @ 2021-09-16 23:44 UTC (permalink / raw)
  To: Marshall Conover; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 5864 bytes --]

I believe that Docker would not have been nearly as successful if shared
libraries had not taken over the Unix world.

Docker would have been helpful for putting together Python and Java
installations, with all their ancillary bits, but less so for C binaries.

Now, though, we run an OS on the machine, put a VM on that, install a
container on that, and run another VM (or JVM) inside the container, then
maybe we're at the level where we can execute actual code. And that misses
all the extra pieces that are distributed and layered and indirected and
virtualized for management, logging, security, data warehousing, ...

It's a wonder any of it works at all.

Here's a map to guide you: https://landscape.cncf.io/

-rob


On Fri, Sep 17, 2021 at 9:15 AM Marshall Conover <marzhall.o@gmail.com>
wrote:

> While I got a chuckle from this, it focuses on security, and I don't
> think security sold docker containers. I think what really sold
> containers was their ability to solve the old, hard problems of
> configuring and maintaining servers.
>
> Docker's use of per-process namespaces is a godsend for running
> different services on the same machine. I no longer run into two
> applications fighting over dependency versions, because both
> applications are running in their own world. This was somewhat
> possible in chroots, but as someone who tried to use chroots that way
> a decade ago, docker's made it trivial.
>
> Containers are also a godsend for applications that have to be
> deployed somewhere else. I know a container I deploy will have
> everything it needs wherever it goes, and will be exactly the thing I
> built and tested. It's hard to understate the benefits of this: when
> deploying, I no longer run into issues like "oh shoot, there was some
> configuration I forgot about on the dev server that I need for prod."
> I no longer have to create server configuration documentation either,
> as the documentation is "docker build," followed by "docker run." When
> we were first starting out on our current project, we built a
> container that runs our build system's agents. At one point the VM on
> which we were running those agents went down, and our stop-gap fix was
> to download and run a few copies of that container locally. As a
> result, we had builds going the entire time we worked to fix the
> issue.
>
> ---------------
>
> Separately, for the larger discussion, I think the
> abstraction-aerospace-engineering seen over the last few decades comes
> from the adage "necessity is the mother of invention." People writing
> business logic today are targeting an OS-independent platform: the
> browser. That's where developers need solutions, and that's where we
> see movement. Considering this, it's no surprise the browser has
> stumbled backwards from a markup language-renderer into a full
> platform for downloading and running applications and managing their
> resources, as well as providing complex abstractions for interacting
> with distributed systems. And it's no surprise those distributed
> systems have separated as much as possible from whatever's not the
> browser.
>
> In fact, we're seeing agreement in the browser ecosystem for problems
> like the directory system choice mentioned above. The OIDC workflow
> was born out of the internet's many-users-to-many-services issue. Now,
> it's such a decided approach for managing users' access to services
> that big names like Amazon and Google offer identity provider services
> using it, and I, as a service writer, can swap between any of them
> transparently. The services I run only care that the token they're
> handed is signed by the auth server they're configured to use, and
> that the token says the user is allowed to use the service contacted.
> The applications I write and use have no clue what the OS' permissions
> are for anything they deal with. For them, OS permissions have been
> made redundant.
>
> With this context, I think most of us here have learned by experience
> why the OS gets no more development, in every discussion they've had
> with management where they've said "we need to refactor some code that
> is wonky, but mostly works, because there will probably be errors and
> bugs and security issues in the future if we don't." Management -
> which in this case, means the world at large - demands new features,
> not unspecified heisen-benefits from redoing things that already work.
> For new features, the browser is their only recourse.
>
> And, to boot - if you change the thing under the browser, what if it
> breaks the browser?
>
> Cheers!
>
> Marshall
>
>
>
>
>
>
>
>
>
>
> On Thu, Sep 16, 2021 at 3:41 PM Larry McVoy <lm@mcvoy.com> wrote:
> >
> > On Thu, Sep 16, 2021 at 12:34:15PM -0700, Jon Steinhart wrote:
> > > As I've said before, I'm having difficulty distinguishing the "full
> stack"
> > > in full stack programming from a compost heap.  It's not OK to me from
> a
> > > security, safety, and reliability perspective to build on a rotting
> > > foundation.
> >
> > Amen.
> >
> > > It's my opinion that the whole container thing sort of started as a "we
> > > can't secure the underlying system so we'll build something secure on
> top"
> > > combined with "it's no fun to fix the unnecessary incompatible mess
> among
> > > virtually identical systems that we've made so we'll build a new fix-it
> > > layer" ideologies.  How long until problems are found with containers
> > > it's decided that the way to fix it is to build "safe deposit boxes"
> that
> > > run in container?  Is there ever an end in sight?
> >
> > I think it is that the newer kids are less willing to understand stuff.
> > So they build something on top that they understand.  I agree that they
> > will hit problems and likely build "safe deposit boxes" because the
> > containers are "too complex".
> >
> > Oh, and get off my lawn!
>

[-- Attachment #2: Type: text/html, Size: 7043 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 19:41     ` Larry McVoy
  2021-09-16 23:14       ` Marshall Conover
@ 2021-09-16 23:45       ` Jon Steinhart
  2021-09-17  0:06         ` Al Kossow
  2021-09-17  0:32         ` Larry McVoy
  2021-09-16 23:54       ` David Arnold
  2 siblings, 2 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-16 23:45 UTC (permalink / raw)
  To: TUHS main list

Larry McVoy writes:
> I think it is that the newer kids are less willing to understand stuff.
> So they build something on top that they understand.  I agree that they
> will hit problems and likely build "safe deposit boxes" because the
> containers are "too complex".

Like usual, we agree on this sort of stuff.  A conundrum for me is that
this stuff that "they understand" is in my opinion way more complicated
than understanding computer hardware and/or an operating system.  So I'm
not sure where the win is.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 19:41     ` Larry McVoy
  2021-09-16 23:14       ` Marshall Conover
  2021-09-16 23:45       ` Jon Steinhart
@ 2021-09-16 23:54       ` David Arnold
  2021-09-17  1:10         ` Jon Steinhart
  2021-09-17 17:39         ` Bakul Shah
  2 siblings, 2 replies; 52+ messages in thread
From: David Arnold @ 2021-09-16 23:54 UTC (permalink / raw)
  To: Larry McVoy; +Cc: TUHS main list

> On 17 Sep 2021, at 05:41, Larry McVoy <lm@mcvoy.com> wrote:

<…>

> I think it is that the newer kids are less willing to understand stuff.
> So they build something on top that they understand.  I agree that they
> will hit problems and likely build "safe deposit boxes" because the
> containers are "too complex”.

Writing a new OS in the 70s or even early 80s meant that you had to replace or port a compiler toolchain, an editor, an email client, a news reader, an IRC client, a couple of simple games, and whatever applications your university/research lab needed to justify its grant money.  It was a chunk of work, but it was surmountable by a small team or even a dedicated individual.  It was demonstrably possible to build your own machine from CPU upward within a reasonable timeframe (eg. Wirth’s Oberon).

It’s still possible (and perhaps even easier) to do that today, but no-one’s really happy with an OS that only provides a handful of applications.  In particular, as has been widely stated, a modern web browser is orders of magnitude more work than an OS.  But expectations for every type of application have moved on, and a modern editor, chat/messaging app, or game is also likely orders of magnitude more complex and feature-rich than what was once acceptable.

For a while, it was possible to have a “POSIX emulation” module/layer/whatever (was Mach the first to go this route?) as a shortcut to this but the breadth of the APIs needed to run, eg. Chrome/ium is again orders of magnitude more work than what was needed to port vi/emacs/rn/etc.

And it’s not just those applications: to have your new OS be useful, you need to support a dozen languages, a hundred protocols, thousands of libraries … a vast range of stuff that would take years, perhaps decades, to port over or reinvent in your new paradigm.

The idea that you’d turn your back on the accumulated value of 50 years of countless people’s work because your set of system calls is slightly better than the one you’ve got now … that’s a very, very big call.

So I think the notion that “the kids” are less willing to understand, or to drill deep, is doing them a disservice.  They do understand, and they (mostly) make the choice to leverage that body of work rather than embark on the futility of starting afresh.





d



^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:45       ` Jon Steinhart
@ 2021-09-17  0:06         ` Al Kossow
  2021-09-17  4:06           ` John Cowan
  2021-09-17  0:32         ` Larry McVoy
  1 sibling, 1 reply; 52+ messages in thread
From: Al Kossow @ 2021-09-17  0:06 UTC (permalink / raw)
  To: tuhs

On 9/16/21 4:45 PM, Jon Steinhart wrote:
> A conundrum for me is that
> this stuff that "they understand" is in my opinion way more complicated
> than understanding computer hardware and/or an operating system.

When a young'un says something like this to snapshot the data in a structure,
I'm convinced my title at CHM should be "20th Century Software Curator" because it
took me an hour to figure out what that sentence even meant.

"You specialize some template for the structures you want to be able to save, and use a concept for a visitor to register members."




^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:45       ` Jon Steinhart
  2021-09-17  0:06         ` Al Kossow
@ 2021-09-17  0:32         ` Larry McVoy
  1 sibling, 0 replies; 52+ messages in thread
From: Larry McVoy @ 2021-09-17  0:32 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Thu, Sep 16, 2021 at 04:45:15PM -0700, Jon Steinhart wrote:
> Larry McVoy writes:
> > I think it is that the newer kids are less willing to understand stuff.
> > So they build something on top that they understand.  I agree that they
> > will hit problems and likely build "safe deposit boxes" because the
> > containers are "too complex".
> 
> Like usual, we agree on this sort of stuff.  A conundrum for me is that
> this stuff that "they understand" is in my opinion way more complicated
> than understanding computer hardware and/or an operating system.  So I'm
> not sure where the win is.

Someone, sorry, I suck at names, I think he is in aerospace or similar,
had a pretty rational view on why docker made things easier.  It was 
today so should be easy to find.  

The part that I don't understand is why it seems so hard to deploy
stuff today.  We supported the same application, a pretty complicated
one, 636K lines of code, on every Unix variant, Linux {32,64} {every
arch including IBM 360}, MacOS {PPC, x86, and I'm working on M1},
Windows {XP..} and it wasn't that hard.  Granted, of the core team,
I'm the least intelligent so I hired well, but still.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 19:27         ` Dan Cross
@ 2021-09-17  0:34           ` Theodore Ts'o
  2021-09-17  0:44             ` Larry McVoy
  2021-09-17  1:33             ` Dan Cross
  0 siblings, 2 replies; 52+ messages in thread
From: Theodore Ts'o @ 2021-09-17  0:34 UTC (permalink / raw)
  To: Dan Cross; +Cc: TUHS main list, Douglas McIlroy

On Thu, Sep 16, 2021 at 03:27:17PM -0400, Dan Cross wrote:
> >
> > I'm really not convinced trying to build distributed computing into
> > the OS ala Plan 9 is viable.
> 
> It seems like plan9 itself is an existence proof that this is possible.
> What it did not present was an existence proof of its scalability and it
> wasn't successful commercially. It probably bears mentioning that that
> wasn't really the point of plan9, though; it was a research system.

I should have been more clear.  I'm not realliy convinced that
building distributed computing into the OS ala Plan 9 is viable from
the perspective of commercial success.  Of course, Plan 9 did it; but
it did it as a research system.

The problem is that if a particular company is convinced that they
want to use Yellow Pages as their directory service --- or maybe X.509
certificates as their authentication system, or maybe Apollo RPC is
the only RPC system for a particularly opinionated site administrator
--- and these prior biases disagree with the choices made by a
particular OS that had distributed computing services built in as a
core part of its functionality, that might be a reason for a
particular customer *not* to deploy a particular distributed OS.

Of course, this doesn't matter if you don't care if anyone uses it
after the paper(s) about said OS has been published.

> Plan 9, as just one example, asked a lot of questions about the issues you
> mentioned above 30 years ago. They came up with _a_ set of answers; that
> set did evolve over time as things progressed. That doesn't mean that those
> questions were resolved definitively, just that there was a group of
> researchers who came up with an approach to them that worked for that group.

There's nothing stopping researchers from creating other research OS's
that try to answer that question.  However, creating an entire new
local node OS from scratch is challenging[1], and then if you then
have to recreate new versions of Kerberos, an LDAP directory server,
etc., so they all of these functions can be tightly integrated into a
single distributed OS ala Plan 9, that seems to be a huge amount of
work, requiring a lot of graduate students to pull off.

[1] http://doc.cat-v.org/bell_labs/utah2000/   (Page 14, Standards)

> What's changed is that we now take for granted that Linux is there, and
> we've stopped asking questions about anything outside of that model.

It's unclear to me that Linux is blamed as the reason why researchers
have stopped asking questions outside of that model.  Why should Linux
have this effect when the presence of Unix didn't?

Or is the argument that it's Linux's fault that Plan 9 has apparently
failed to compete with it in the marketplace of ideas?  And arguably,
Plan 9 failed to make headway against Unix (and OSF/DCE, and Sun NFS,
etc.) in the early to mid 90's, which is well before Linux's became
popular, so that argument doesn't really make sense, either.

						- Ted

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:44         ` Rob Pike
@ 2021-09-17  0:37           ` Larry McVoy
  0 siblings, 0 replies; 52+ messages in thread
From: Larry McVoy @ 2021-09-17  0:37 UTC (permalink / raw)
  To: Rob Pike; +Cc: TUHS main list

On Fri, Sep 17, 2021 at 09:44:42AM +1000, Rob Pike wrote:
> Now, though, we run an OS on the machine, put a VM on that, install a
> container on that, and run another VM (or JVM) inside the container, then
> maybe we're at the level where we can execute actual code. And that misses
> all the extra pieces that are distributed and layered and indirected and
> virtualized for management, logging, security, data warehousing, ...
> 
> It's a wonder any of it works at all.

I think a huge part of why it works is Pugs exporting the IOMMU to
user space.  If I understand that work correctly, it really doesn't
matter how many layers you have, if the layers use what he did, then
the layers don't matter for I/O and I/O is usually where things get slow.

Rob, that link you sent makes my head spin.  Which was probably your intent.

--lm

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  0:34           ` Theodore Ts'o
@ 2021-09-17  0:44             ` Larry McVoy
  2021-09-17 17:07               ` Bakul Shah
  2021-09-17  1:33             ` Dan Cross
  1 sibling, 1 reply; 52+ messages in thread
From: Larry McVoy @ 2021-09-17  0:44 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Douglas McIlroy

On Thu, Sep 16, 2021 at 08:34:52PM -0400, Theodore Ts'o wrote:
> > What's changed is that we now take for granted that Linux is there, and
> > we've stopped asking questions about anything outside of that model.
> 
> It's unclear to me that Linux is blamed as the reason why researchers
> have stopped asking questions outside of that model.  Why should Linux
> have this effect when the presence of Unix didn't?

Linux runs on _everything_.  From your phone to the top 500 HPC clusters.

Unix, for all that it did, never had that level of success.  I credit
Unix for a lot, including showing Linux what it should be, but Linux
took that model and ran with it.

Plan 9 is very cool but I am channeling my inner Clem, Plan 9 didn't meet
Clem's law.  It was never compelling enough to make the masses love it.
Linux was good enough.

We can argue about if that is a good thing or not, I've watched Linux
become more complex and seen docker et al react to that.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:54       ` David Arnold
@ 2021-09-17  1:10         ` Jon Steinhart
  2021-09-17  1:28           ` Larry McVoy
  2021-09-17  2:48           ` Theodore Ts'o
  2021-09-17 17:39         ` Bakul Shah
  1 sibling, 2 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17  1:10 UTC (permalink / raw)
  To: TUHS main list

David Arnold writes:
> And it’s not just those applications: to have your new OS be useful,
> you need to support a dozen languages, a hundred protocols, thousands of
> libraries … a vast range of stuff that would take years, perhaps decades,
> to port over or reinvent in your new paradigm.
> 
> The idea that you’d turn your back on the accumulated value of 50 years
> of countless people’s work because your set of system calls is slightly
> better than the one you’ve got now … that’s a very, very big call.
> 
> So I think the notion that “the kids” are less willing to understand,
> or to drill deep, is doing them a disservice.  They do understand, and
> they (mostly) make the choice to leverage that body of work rather than
> embark on the futility of starting afresh.

I have to respectfully disagree which is a challenge because being
disagreeable comes more naturally than being respectful :-)

We already have this.  I kind of wonder what actual value could have been
created with the resources that went into supporting the dozen languages,
hundred protocols, and so on.  Is there value to me that a library exists
that lets me to something in python that is identical to the library that
lets me do that same thing in perl or the one that lets me do it in php or
the one that lets me do it in ...?  No.

You made a big assumption that I was suggesting tossing prior work and API
specs which I wasn't.  Would you have wanted to have the 32 bit system call
API frozen because it worked and not wanted 64 bit versions?  History shows
plenty of good work going into compatibility when the underlying technology
evolves.

Don't know how much time you spend with "the kids" these days.  I make it a
point to do so when I can; SIGCOVID has cut into that unfortunately.  One
can get a CS degree without ever taking an OS course at many respectable
institutions.  Many are not so much making a choice as doing what they can
which is inadequate in my opinion.

Was discussing this with someone the other day.  I'm glad that I have an
engineering degree instead of a computer science degree.  And I'm also glad
that I prevailed with one of my mentees to double major in EE and CS when
he wanted to bail on the EE.  While it's a generalization, as an engineeer I
was educated on how the universe worked - chemistry, physics, and so on.  It
was up to me to figure out how to apply that knowledge - I wasn't taught how
to lay out a circuit board or to use a CAD package or to write an app.  A
modern CS degree at many institutions is vocational training in programming.
It's not the same thing.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  1:10         ` Jon Steinhart
@ 2021-09-17  1:28           ` Larry McVoy
  2021-09-17  1:40             ` Jon Steinhart
  2021-09-17  2:48           ` Theodore Ts'o
  1 sibling, 1 reply; 52+ messages in thread
From: Larry McVoy @ 2021-09-17  1:28 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Thu, Sep 16, 2021 at 06:10:11PM -0700, Jon Steinhart wrote:
> Was discussing this with someone the other day.  I'm glad that I have an
> engineering degree instead of a computer science degree.  

I have and BS and MS in computer science, with most of a minor in
systems arch from the EE department (but not all, I did it for fun,
I should have finished it).

That said, when people ask me what I am, I say I'm an engineer.  And 
proud of it, I love being an engineer, to me it just means you are 
someone who figures stuff out.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  0:34           ` Theodore Ts'o
  2021-09-17  0:44             ` Larry McVoy
@ 2021-09-17  1:33             ` Dan Cross
  1 sibling, 0 replies; 52+ messages in thread
From: Dan Cross @ 2021-09-17  1:33 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: TUHS main list, Douglas McIlroy

[-- Attachment #1: Type: text/plain, Size: 6825 bytes --]

On Thu, Sep 16, 2021 at 8:34 PM Theodore Ts'o <tytso@mit.edu> wrote:

> On Thu, Sep 16, 2021 at 03:27:17PM -0400, Dan Cross wrote:
> > >
> > > I'm really not convinced trying to build distributed computing into
> > > the OS ala Plan 9 is viable.
> >
> > It seems like plan9 itself is an existence proof that this is possible.
> > What it did not present was an existence proof of its scalability and it
> > wasn't successful commercially. It probably bears mentioning that that
> > wasn't really the point of plan9, though; it was a research system.
>
> I should have been more clear.  I'm not realliy convinced that
> building distributed computing into the OS ala Plan 9 is viable from
> the perspective of commercial success.  Of course, Plan 9 did it; but
> it did it as a research system.
>
> The problem is that if a particular company is convinced that they
> want to use Yellow Pages as their directory service --- or maybe X.509
> certificates as their authentication system, or maybe Apollo RPC is
> the only RPC system for a particularly opinionated site administrator
> --- and these prior biases disagree with the choices made by a
> particular OS that had distributed computing services built in as a
> core part of its functionality, that might be a reason for a
> particular customer *not* to deploy a particular distributed OS.
>

Ah, I take your meaning. Yes, I can see that being a problem. But we've had
similar problems before: "we only buy IBM", or, "does it integrate into our
VAXcluster?" Put another way, _every_ system has opinions about how to do
things. I suppose the distinction you're making is that we can paper over
so many of those by building abstractions on top of the "node" OS. But the
node OS is already forcing a shape onto our solutions. Folks working on the
Go runtime have told me painful stories about detection of blocking system
calls using timers and signals: wouldn't it be easier if the system
provided real asynchronous abstractions? But the system call model in
Unix/Linux/plan9 etc is highly synchronous. If `open` takes a while for
whatever reason (say, blocking on reading directory entries looking up a
name?) there's no async IO interface for that, hence shenanigans. But
that's what the local node gives me; c'est la vie.

Of course, this doesn't matter if you don't care if anyone uses it
> after the paper(s) about said OS has been published.
>

I suspect most researchers don't expect the actual research artifacts to
make it directly into products, but that the ideas will hopefully have some
impact. Interestingly, Unix seems to have been an exception to this in that
Unix itself did make it into industry.

> Plan 9, as just one example, asked a lot of questions about the issues you
> > mentioned above 30 years ago. They came up with _a_ set of answers; that
> > set did evolve over time as things progressed. That doesn't mean that
> those
> > questions were resolved definitively, just that there was a group of
> > researchers who came up with an approach to them that worked for that
> group.
>
> There's nothing stopping researchers from creating other research OS's
> that try to answer that question.


True, but they aren't. I suspect there are a number of confounding factors
at play here; certainly, the breadth and size of the standards they have to
implement is an issue, but so is lack of documentation. No one is seriously
looking at new system architectures, though.


> However, creating an entire new
> local node OS from scratch is challenging[1], and then if you then
> have to recreate new versions of Kerberos, an LDAP directory server,
> etc., so they all of these functions can be tightly integrated into a
> single distributed OS ala Plan 9, that seems to be a huge amount of
> work, requiring a lot of graduate students to pull off.
>
> [1] http://doc.cat-v.org/bell_labs/utah2000/   (Page 14, Standards)
>

Yup. That is the presentation I meant when I mentioned Rob Pike lamenting
the situation 20 years ago in the previous message and earlier in the
thread.

An interesting thing here is that we assume that we have to redo _all_ of
that, though. A lot of the software out there is just code that does
something interesting, but actually touches the system in a pretty small
way. gvisor is an interesting example of this; it provides something that
looks an awful lot like Linux to an application, and a lot of stuff can run
under it. But the number of system calls _it_ in turn makes to the
underlying system is much smaller.

> What's changed is that we now take for granted that Linux is there, and
> > we've stopped asking questions about anything outside of that model.
>
> It's unclear to me that Linux is blamed as the reason why researchers
> have stopped asking questions outside of that model.  Why should Linux
> have this effect when the presence of Unix didn't?
>

a) There's a lot more Linux in the world than there ever was Unix. b) There
are more computers now than there were when Unix was popular. c) computers
are significantly more complex now than they were when Unix was written.

But to be clear, I don't think this trend started with Linux; I get the
impression that by the 1980s, a lot of research focused on a Unix-like
model to the exclusion of other architectures. The PDP-10 was basically
dead by 1981, and we haven't seen a system like TOPS-20 since the 70s.

Or is the argument that it's Linux's fault that Plan 9 has apparently
> failed to compete with it in the marketplace of ideas?


It's hard to make that argument when Linux borrowed so many of plan9's
ideas: /proc, per-process namespaces, etc.


> And arguably,
> Plan 9 failed to make headway against Unix (and OSF/DCE, and Sun NFS,
> etc.) in the early to mid 90's, which is well before Linux's became
> popular, so that argument doesn't really make sense, either.
>

That wasn't the argument. There are a number of reasons why plan9 failed to
achieve commercial success relative to Unix; most of them have little to do
with technology. In many ways, AT&T strangled the baby by holding it too
tightly to its chest, fearful of losing control the way they "lost" control
of Unix (ironically, something that allowed Unix to flourish and become
wildly successful). Incompatibility with the rest of the world was likely
an issue, but inaccessibility and overly restrictive licensing in the early
90s practically made it a foregone conclusion.

Also, it's a little bit of an aside, but I think we often undercount the
impact of individual preference on systems. In so many ways, Linux
succeeded because, simply put, people liked working on Linux more than they
liked working on other systems. You've mentioned yourself that it was more
fun to hack on Linux without having to appease some of the big
personalities in the BSD world.

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 8722 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:14       ` Marshall Conover
  2021-09-16 23:44         ` Rob Pike
@ 2021-09-17  1:38         ` Jon Steinhart
  2021-09-17  3:54         ` John Cowan
  2 siblings, 0 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17  1:38 UTC (permalink / raw)
  To: TUHS main list

Marshall Conover writes:
>
> Separately, for the larger discussion, I think the
> abstraction-aerospace-engineering seen over the last few decades comes
> from the adage "necessity is the mother of invention." People writing
> business logic today are targeting an OS-independent platform: the
> browser.

Wow.  I think that it would be more accurate to say that people writing
business logic today are targeting the browser because other people
are going through the trouble of porting it to different platforms.
Doesn't seem to me the best use of resources given that browsers are
more complex than operating systems.  And I've had many an experience
with things that are not portable among browsers.  Of course, given
that firefox is trying to die by regularly alienating users and that to
a first approximation much everything else is chrome or chrome based,
you're effectively saying that people are targeting a single operating
system even though we don't call a brower an OS.

And while there's no going back, I think that browsers suck.  Doesn't seem
that anybody had the foresight in the early days to realize that they
were really building a virtual machine, so the one that we have ended up
with is a Rube Goldberg contraption.

CSS is one of the brower components that I find especially appalling.
I understand its genesis and all that.  Would be lovely to be able to
make stuff just work without having to "program".  Despite revision after
revision it's still impossible to lay out things as desired without
reverting to JavaScript.  While it didn't start that way, at this
point there are so many properties with twisty and often undocumented
interactions that it would be better to toss it and just write programs.
Of course, programming is "hard" and it's supposedly easier to pore
though online forums looking for answers to questions like "I think that
this can be done but I have no idea how so can someone please share an
incantation with me.  I personally prefer a language that has a small
number of primitives that can be combined in an understandable manner
than a huge number of poorly documented primitives that no one person
fully understands.  And don't tell me that JavaScript is the answer; while
it has some good, it suffers from being the dumping ground for people
who were never able to get their favorite feature into other languages;
it's an incoherent mess.

I know that Larry and Clem and I agree on the value of past work.  I was a
proofreader for Alan Wirf-Brock's 20 years of JavaScript article.  I was
busy with other stuff when JavaScript began so wasn't familiar with some
of the history.  Kind of shook my head reading about Eich's big week-long
sprint to get the parser up and running.  Though to myself that it would
have only been a half-day sprint at most had he used existing tools such
as lex and yacc, and had he done so we wouldn't still be suffering from
the optional semicolon problem 20 years later.

Don't mean to offend anybody here, all just my opinion.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  1:28           ` Larry McVoy
@ 2021-09-17  1:40             ` Jon Steinhart
  2021-09-17  2:04               ` Larry McVoy
  0 siblings, 1 reply; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17  1:40 UTC (permalink / raw)
  To: TUHS main list

Larry McVoy writes:
> On Thu, Sep 16, 2021 at 06:10:11PM -0700, Jon Steinhart wrote:
> > Was discussing this with someone the other day.  I'm glad that I have an
> > engineering degree instead of a computer science degree.  
>
> I have and BS and MS in computer science, with most of a minor in
> systems arch from the EE department (but not all, I did it for fun,
> I should have finished it).
>
> That said, when people ask me what I am, I say I'm an engineer.  And 
> proud of it, I love being an engineer, to me it just means you are 
> someone who figures stuff out.

I'm the opposite.  Would have double-majored in EE and CS but my school
wouldn't allow double majoring unless you stayed for 9 semesters.  I
finished my EE in 7 semesters and was 2 classes shy of the CS degree
and couldn't justify another year just for 2 classes.  In hindsight
I'm not sure that that was the correct choice because I would have
partied my ass off.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  1:40             ` Jon Steinhart
@ 2021-09-17  2:04               ` Larry McVoy
  2021-09-17  2:21                 ` Jon Steinhart
  0 siblings, 1 reply; 52+ messages in thread
From: Larry McVoy @ 2021-09-17  2:04 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Thu, Sep 16, 2021 at 06:40:47PM -0700, Jon Steinhart wrote:
> Larry McVoy writes:
> > On Thu, Sep 16, 2021 at 06:10:11PM -0700, Jon Steinhart wrote:
> > > Was discussing this with someone the other day.  I'm glad that I have an
> > > engineering degree instead of a computer science degree.  
> >
> > I have and BS and MS in computer science, with most of a minor in
> > systems arch from the EE department (but not all, I did it for fun,
> > I should have finished it).
> >
> > That said, when people ask me what I am, I say I'm an engineer.  And 
> > proud of it, I love being an engineer, to me it just means you are 
> > someone who figures stuff out.
> 
> I'm the opposite.  Would have double-majored in EE and CS but my school
> wouldn't allow double majoring unless you stayed for 9 semesters.  I
> finished my EE in 7 semesters and was 2 classes shy of the CS degree
> and couldn't justify another year just for 2 classes.  In hindsight
> I'm not sure that that was the correct choice because I would have
> partied my ass off.

You do you, I had undiagnosed ADD so partys in college were just an 
awkward cringe fest for me.  Looking backwards, now that I've figured
that out, yeah, I can see it, sort of, if I knew then what I know now.

I was pretty committed to learning in college.  I'm not trying to judge
or anything, it was just such a fun focussed time for me, I'd happily
give up a party (where I was gonna get nothing) for a few hours on slovax
where the BSD source was.

Details aside, I think we both self identify as engineers and love it.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  2:04               ` Larry McVoy
@ 2021-09-17  2:21                 ` Jon Steinhart
  0 siblings, 0 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17  2:21 UTC (permalink / raw)
  To: TUHS main list

Larry McVoy writes:
>
> You do you, I had undiagnosed ADD so partys in college were just an 
> awkward cringe fest for me.  Looking backwards, now that I've figured
> that out, yeah, I can see it, sort of, if I knew then what I know now.
>
> I was pretty committed to learning in college.  I'm not trying to judge
> or anything, it was just such a fun focussed time for me, I'd happily
> give up a party (where I was gonna get nothing) for a few hours on slovax
> where the BSD source was.
>
> Details aside, I think we both self identify as engineers and love it.

OK, we're getting pretty far afield from UNIX.  Probably wasn't using the
word "party" in the traditional sense - of the drugs, sex, and rock-n-roll
college was consumed by the last two.  BSD wasn't a thing when I was in
school; I do have a memory of coming in to work at BTL after dinner one
evening when Ken was walking out the door with mag tapes under his arm on
his way to Berkeley for sabbatical.  I got to play with computers and UNIX
at BTL, college was building and operating radio stations.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  1:10         ` Jon Steinhart
  2021-09-17  1:28           ` Larry McVoy
@ 2021-09-17  2:48           ` Theodore Ts'o
  1 sibling, 0 replies; 52+ messages in thread
From: Theodore Ts'o @ 2021-09-17  2:48 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Thu, Sep 16, 2021 at 06:10:11PM -0700, Jon Steinhart wrote:
> 
> You made a big assumption that I was suggesting tossing prior work and API
> specs which I wasn't.  Would you have wanted to have the 32 bit system call
> API frozen because it worked and not wanted 64 bit versions?  History shows
> plenty of good work going into compatibility when the underlying technology
> evolves.

Unfortunately, Implementing all of the old API specs is not easy if
you're starting from scratch as you create your new OS.  As David
wrote in an earlier paragraph:

> For a while, it was possible to have a “POSIX emulation”
> module/layer/whatever (was Mach the first to go this route?) as a
> shortcut to this but the breadth of the APIs needed to run,
> eg. Chrome/ium is again orders of magnitude more work than what was
> needed to port vi/emacs/rn/etc.

And this is the same observation Rob Pike made in his cri du coeur in
2000:

  * To be a viable computer system, one must honor a huge list of
    large, and often changing, standards: TCP/IP, HTTP, HTML, XML,
    CORBA, Unicode, POSIX, NFS, SMB, MIME, POP, IMAP, X, ...

  * A huge amount of work, but if you don't honor the standards,
    you're marginalized.

  * I estimate that 90-95% of the work in Plan 9 was directly or
    indirectly to honor externally imposed standards.

> Don't know how much time you spend with "the kids" these days.  I make it a
> point to do so when I can; SIGCOVID has cut into that unfortunately.  One
> can get a CS degree without ever taking an OS course at many respectable
> institutions.  Many are not so much making a choice as doing what they can
> which is inadequate in my opinion.
> 
> Was discussing this with someone the other day.  I'm glad that I have an
> engineering degree instead of a computer science degree.  And I'm also glad
> that I prevailed with one of my mentees to double major in EE and CS when
> he wanted to bail on the EE.  While it's a generalization, as an engineeer I
> was educated on how the universe worked - chemistry, physics, and so on.  It
> was up to me to figure out how to apply that knowledge - I wasn't taught how
> to lay out a circuit board or to use a CAD package or to write an app.  A
> modern CS degree at many institutions is vocational training in programming.
> It's not the same thing.

When I studied CS in the late 80's, MIT's EE/CS department required
all EE's and CS's to take 2 foundational CS classes, and 2
foundational EE classes.  This meant that CS folks needed to
understand how to build op-amps from transitors (6.002) and descrete
and continuous FFT's (6.003).  And EE folks needed to be able
understand Lambda calculus (6.001), and to build stack and general
register computers using 74XX TTL chips on a breadbroad (6.004).
Furthermore, CS students needed to take an OS course and/or a compiler
course, so by the time you graduated, you understood computers from a
"full stack" perspective --- from transitors, to AND/OR gates, to CPU
design, to compilers, to OS's, to systems issues around desining big
systems like Multics and the SABRE Airline Reservations systems.

These days, at MIT, one of things students are taught is how figure
out what an under-documented abstraction (implemented in Python),
partially by reading the code (but it's way too complex), so it's
mostly by deducing the abstraction by running experiments on the
Python Library code in question.  Given how complex computers have
gotten, that's probably more realistic anticipation of what students
will need once they graduate, but yeah, it does seem a bit like
"vocational training in programming".  And that's quite a change.

When I was an undergraduate, MIT was proud of the fact that they
didn't teach CS students the C language; after all, that would be
*way* too practical/vocational.  The assumption was after you learned
Scheme and CLU, you'd be able pick up other languages on the fly.
(And they didn't really *teach* Scheme per se; the first lecture in
6.001 was about the Lambda calculus, and if you couldn't figure out
Scheme syntax from the reference manual so you could do the first
problem set, well, the EE/CS department was heavily over-subscribed
anyway, and it was a good way of filtering out the less committed
undergraduates.  :-)

      	       	  	    	- Ted

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:14       ` Marshall Conover
  2021-09-16 23:44         ` Rob Pike
  2021-09-17  1:38         ` Jon Steinhart
@ 2021-09-17  3:54         ` John Cowan
  2 siblings, 0 replies; 52+ messages in thread
From: John Cowan @ 2021-09-17  3:54 UTC (permalink / raw)
  To: Marshall Conover; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Thu, Sep 16, 2021 at 7:15 PM Marshall Conover <marzhall.o@gmail.com>
wrote:


> I know a container I deploy will have
> everything it needs wherever it goes, and will be exactly the thing I
> built and tested.


Up to a point, Minister.  You can mount a scratch monkey, but you can't
mount a scratch Internet or a scratch AWS or a scratch Google.

> At one point the VM on
> which we were running those agents went down, and our stop-gap fix was
> to download and run a few copies of that container locally.
>

That's true if the container isn't too vulgar big.  I can run $EMPLOYER's
whole application on my laptop in the root environment, but running it in
Docker is too costly even though that's how it's deployed on AWS.


> from the adage "necessity is the mother of invention." People writing
> business logic today are targeting an OS-independent platform: the
> browser.


Most actual business logic is still in the back end, at least at my part of
the coal face.  The browser is more of a programmable platform as time goes
by, but it's still a Blit even if no longer just a 3270.


> Management -
> which in this case, means the world at large - demands new features,
> not unspecified heisen-benefits from redoing things that already work.
>

There is a pressure toward that.  But when $CLIENTS (who are a lot bigger
than $EMPLOYER) start to complain about how often the application they are
paying $$$$$$$ for falls over due to lack of robustness, things change.
Not everything can be startup-grade.

[-- Attachment #2: Type: text/html, Size: 3245 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  0:06         ` Al Kossow
@ 2021-09-17  4:06           ` John Cowan
  2021-09-17  4:18             ` Al Kossow
  0 siblings, 1 reply; 52+ messages in thread
From: John Cowan @ 2021-09-17  4:06 UTC (permalink / raw)
  To: Al Kossow; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

On Thu, Sep 16, 2021 at 8:13 PM Al Kossow <aek@bitsavers.org> wrote:


> When a young'un says something like this to snapshot the data in a
> structure,
> I'm convinced my title at CHM should be "20th Century Software Curator"
> because it
> took me an hour to figure out what that sentence even meant.
>
> "You specialize some template for the structures you want to be able to
> save, and use a concept for a visitor to register members."
>

Y'know, the guy who invented that jargon is a younker of 70 (and from the
Labs at that).

[-- Attachment #2: Type: text/html, Size: 1096 bytes --]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  4:06           ` John Cowan
@ 2021-09-17  4:18             ` Al Kossow
  0 siblings, 0 replies; 52+ messages in thread
From: Al Kossow @ 2021-09-17  4:18 UTC (permalink / raw)
  To: TUHS main list

On 9/16/21 9:06 PM, John Cowan wrote:

> Y'know, the guy who invented that jargon is a younker of 70 (and from the Labs at that).

and we had a version of "Talking Moose" called "Talking Barney" at Apple in the late 80's
making fun of him and his language.

The tuft of hair on top would pop up when he said something exceptional.

I've not been able to find a copy of it on my backups, sadly.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 19:34   ` Jon Steinhart
  2021-09-16 19:41     ` Larry McVoy
@ 2021-09-17 15:56     ` Bakul Shah
  2021-09-17 18:24       ` ron minnich
  1 sibling, 1 reply; 52+ messages in thread
From: Bakul Shah @ 2021-09-17 15:56 UTC (permalink / raw)
  To: TUHS main list

On Sep 16, 2021, at 12:34 PM, Jon Steinhart <jon@fourwinds.com> wrote:
> 
> It's my opinion that the whole container thing sort of started as a "we
> can't secure the underlying system so we'll build something secure on top"
> combined with "it's no fun to fix the unnecessary incompatible mess among
> virtually identical systems that we've made so we'll build a new fix-it
> layer" ideologies.  How long until problems are found with containers
> it's decided that the way to fix it is to build "safe deposit boxes" that
> run in container?  Is there ever an end in sight?

Recall that previously sysadmins used programs such as ghost to image a
system. A completely operational system with all the required software
could be created fairly quickly with minimum configuration. If your
h/w crashed, you can get up and running fairly quickly on a new machine
(provided your unique bits were backed up & restored). The same thing
could be done for server machines. By minimizing differences you can
apply security patches or put new machines in service quickly. A server
machine needs much more than the main service program before it can
be put in real service but machines providing the same service need
pretty much the same things.

When VMs and containers started getting used, the same model could
be used for provisioning them. The docker folks simplified this
further. Now you can spin up new servers almost trivially (even if
later tooling via Kubernetes and such got quite complicated). Seems
to me, this provisioning of whole systems is what users of technologies
such as jail never quite got it.

A couple of points on this: 1) I think this can be simplified even
further if one can rely on a fast control plane connection by basically
lazily pulling in the contents of each container. 2) If the underlying
system provides a capability architecture, you can probably achieve the
exact same functionality without containers as the required "many worlds"
functionality is already built in. 

-- Bakul

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17  0:44             ` Larry McVoy
@ 2021-09-17 17:07               ` Bakul Shah
  0 siblings, 0 replies; 52+ messages in thread
From: Bakul Shah @ 2021-09-17 17:07 UTC (permalink / raw)
  To: Larry McVoy; +Cc: TUHS main list, Douglas McIlroy

On Sep 16, 2021, at 5:44 PM, Larry McVoy <lm@mcvoy.com> wrote:
> 
> On Thu, Sep 16, 2021 at 08:34:52PM -0400, Theodore Ts'o wrote:
>>> What's changed is that we now take for granted that Linux is there, and
>>> we've stopped asking questions about anything outside of that model.
>> 
>> It's unclear to me that Linux is blamed as the reason why researchers
>> have stopped asking questions outside of that model.  Why should Linux
>> have this effect when the presence of Unix didn't?
> 
> Linux runs on _everything_.  From your phone to the top 500 HPC clusters.
> 
> Unix, for all that it did, never had that level of success.  I credit
> Unix for a lot, including showing Linux what it should be, but Linux
> took that model and ran with it.
> 
> Plan 9 is very cool but I am channeling my inner Clem, Plan 9 didn't meet
> Clem's law.  It was never compelling enough to make the masses love it.
> Linux was good enough.

Things might have been different if Plan9 was open sourced in the same time
frame as 386BSD and early Linux. Back then Linux was not good enough. plan9
was not all that different from unix and simpler to understand and use.

> We can argue about if that is a good thing or not, I've watched Linux
> become more complex and seen docker et al react to that.

As Marshall Conover (& later I) said, containers & docker filled a genuine
need. Not much to do with the complexity of Linux. Plan9 could've provided
a much simpler and cleaner platform than linux but that was not to be.
[plan has a couple of global names spaces that might have needed changing.
Specifically the process id and '#' driver spaces]


-- Bakul


^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-16 23:54       ` David Arnold
  2021-09-17  1:10         ` Jon Steinhart
@ 2021-09-17 17:39         ` Bakul Shah
  2021-09-17 17:51           ` Jon Steinhart
  1 sibling, 1 reply; 52+ messages in thread
From: Bakul Shah @ 2021-09-17 17:39 UTC (permalink / raw)
  To: David Arnold; +Cc: TUHS main list

On Sep 16, 2021, at 4:54 PM, David Arnold <davida@pobox.com> wrote:
> 
> And it’s not just those applications: to have your new OS be useful, you need to support a dozen languages, a hundred protocols, thousands of libraries … a vast range of stuff that would take years, perhaps decades, to port over or reinvent in your new paradigm.
> 
> The idea that you’d turn your back on the accumulated value of 50 years of countless people’s work because your set of system calls is slightly better than the one you’ve got now … that’s a very, very big call.
> 
> So I think the notion that “the kids” are less willing to understand, or to drill deep, is doing them a disservice.  They do understand, and they (mostly) make the choice to leverage that body of work rather than embark on the futility of starting afresh.

I have mixed feelings about this. Unix didn't "throw away"
the mainframe world of computing. It simply created a new
ecosystem, more suited for the microprocessor age. For IBM it
was perhaps the classic Innovator's Dilemma. Similarly now we
have (mostly) the Linux ecosystem, while the actual hardware
has diverged a lot from the C memory model. There are
security issues. There is firmware running on these system
about which the OS knows nothing. We have processors like
Esperanto Tech's 1088 64 bit Risc-V cores, each with its own
vector/tensor unit, 160MB onchip sram and 23.8B transistors
but can take only limited advantage of it. We have super
performant GPUs but programming them is vendor dependent and
a pain. If someone can see a clear path through all this,
and create a new software system, they will simply generate a
new ecosystem and not worry about 50 years worth of work.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 17:39         ` Bakul Shah
@ 2021-09-17 17:51           ` Jon Steinhart
  2021-09-17 18:07             ` Larry McVoy
  2021-09-17 18:34             ` Bakul Shah
  0 siblings, 2 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17 17:51 UTC (permalink / raw)
  To: TUHS main list

Bakul Shah writes:
> I have mixed feelings about this. Unix didn't "throw away"
> the mainframe world of computing. It simply created a new
> ecosystem, more suited for the microprocessor age. For IBM it
> was perhaps the classic Innovator's Dilemma. Similarly now we
> have (mostly) the Linux ecosystem, while the actual hardware
> has diverged a lot from the C memory model. There are
> security issues. There is firmware running on these system
> about which the OS knows nothing. We have processors like
> Esperanto Tech's 1088 64 bit Risc-V cores, each with its own
> vector/tensor unit, 160MB onchip sram and 23.8B transistors
> but can take only limited advantage of it. We have super
> performant GPUs but programming them is vendor dependent and
> a pain. If someone can see a clear path through all this,
> and create a new software system, they will simply generate a
> new ecosystem and not worry about 50 years worth of work.

You're kind of reminding me of the HEP (heterogeneous element
processor) talk that I saw at I think Usenix in Santa Monica.
My opinion is that it was a "kitchen sink" project - let's
throw in a few of these and a few of those and so on.  Also
analogous to what I saw in the housing market up here when
people started cashing in their California huts for Oregon
mansions - when we lived in California we could afford two
columns out front but now we can afford 6 columns, 8 poticos,
6 dormers, 4 turrets, and so on.  Just because you can built
it doesn't keep it from being an ugly mess.

So my question on many of these processors is, has anybody
given any thought to system architecture?  Most likely all
of us have had to suffer with some piece of spiffy hardware
that was pretty much unprogrammable.  Do the performance
numbers mean anything if they can't be achieved in an actual
system configuration?

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 17:51           ` Jon Steinhart
@ 2021-09-17 18:07             ` Larry McVoy
  2021-09-17 21:03               ` Derek Fawcus
  2021-09-17 18:34             ` Bakul Shah
  1 sibling, 1 reply; 52+ messages in thread
From: Larry McVoy @ 2021-09-17 18:07 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Fri, Sep 17, 2021 at 10:51:38AM -0700, Jon Steinhart wrote:
> So my question on many of these processors is, has anybody
> given any thought to system architecture?  Most likely all
> of us have had to suffer with some piece of spiffy hardware
> that was pretty much unprogrammable.  Do the performance
> numbers mean anything if they can't be achieved in an actual
> system configuration?

Apple, much to my surprise, has been thinking about this stuff.
The M1 is one of the better thought out chips in a long time.
I especially liked the idea of slow (and low battery draw)
processors for some apps, and fast ones for apps that need
fast.  It's an obvious idea in retrospect but I hadn't thought
to do it that way.  Clever.

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 15:56     ` Bakul Shah
@ 2021-09-17 18:24       ` ron minnich
  0 siblings, 0 replies; 52+ messages in thread
From: ron minnich @ 2021-09-17 18:24 UTC (permalink / raw)
  To: TUHS main list

As a student weekend operator at Dupont, I saw the last 2 weeks of the
705 vacuum tube machine that ran payroll and then, over two years: the
7080 that ran the 705 in emulation; the 360 that ran the 7080 that ran
the 705; the 370 that ran the 360 that ran the 7800 that ran the 705.
The 370 sacrificed ASCII support so it could emulate a 360 (IBM ran
out of bits in the PSW, so ASCII had to go).

The Blue Gene compute node kernel (CNK) started out supporting the
"most commonly used" 80 or so linux system calls; that grew by about 1
system call a month. As CNK became asymptotic to Linux, some sites
threw in the towel and ran linux. Frequently, emulation is not enough.

Linux itself supported SCO for a time; freebsd has a good linux
subsystem; one could argue that Linux emulates Linux, given its binary
compatibility guarantee.

Unix had a JCL command through v5 and still has a GECOS field in the password.

The x86 allocates 1/256 of its gigantic opcode space to the DAA
instruction nobody uses. A multi-hundred core server can still boot
CP/M and DOS 1.0

RISC-V started as clean-ish break with the past, and has followed the
standard evilutionary path to x86-inspired 4k page sizes; big-endian
support; and runtime BIOS.

Successful systems rarely come as a complete break with the past.

People frequently expect that the Unix "clean break" experience can be
repeated, but. if anything, the history of Unix shows why that
experience is unlikely to be repeated.

But if you're going to do a new kernel, and you want broad usage,
you're going to provide a VM to run Linux or something like it, or
you're going to fail.

Or target something invisible, like a security token, then you are
free to do what you will -- the Google Open Titan security token runs
a kernel written in Rust. Nobody knows about it so nobody cares.

ron

p.s. I remember the person who finally ported payroll from the 705
binary. "There's a bug in there, maybe, but i can't find it" -- 2
weeks later, in a not-to-be-repeated occurence, the paycheck of all us
weekend students was 2x normal. We couldn't figure out who to report
it to so we let it slide.
p.p.s. The 7080 was the last mainframe I worked with that smelled of
machine oil.

On Fri, Sep 17, 2021 at 8:57 AM Bakul Shah <bakul@iitbombay.org> wrote:
>
> On Sep 16, 2021, at 12:34 PM, Jon Steinhart <jon@fourwinds.com> wrote:
> >
> > It's my opinion that the whole container thing sort of started as a "we
> > can't secure the underlying system so we'll build something secure on top"
> > combined with "it's no fun to fix the unnecessary incompatible mess among
> > virtually identical systems that we've made so we'll build a new fix-it
> > layer" ideologies.  How long until problems are found with containers
> > it's decided that the way to fix it is to build "safe deposit boxes" that
> > run in container?  Is there ever an end in sight?
>
> Recall that previously sysadmins used programs such as ghost to image a
> system. A completely operational system with all the required software
> could be created fairly quickly with minimum configuration. If your
> h/w crashed, you can get up and running fairly quickly on a new machine
> (provided your unique bits were backed up & restored). The same thing
> could be done for server machines. By minimizing differences you can
> apply security patches or put new machines in service quickly. A server
> machine needs much more than the main service program before it can
> be put in real service but machines providing the same service need
> pretty much the same things.
>
> When VMs and containers started getting used, the same model could
> be used for provisioning them. The docker folks simplified this
> further. Now you can spin up new servers almost trivially (even if
> later tooling via Kubernetes and such got quite complicated). Seems
> to me, this provisioning of whole systems is what users of technologies
> such as jail never quite got it.
>
> A couple of points on this: 1) I think this can be simplified even
> further if one can rely on a fast control plane connection by basically
> lazily pulling in the contents of each container. 2) If the underlying
> system provides a capability architecture, you can probably achieve the
> exact same functionality without containers as the required "many worlds"
> functionality is already built in.
>
> -- Bakul

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 17:51           ` Jon Steinhart
  2021-09-17 18:07             ` Larry McVoy
@ 2021-09-17 18:34             ` Bakul Shah
  2021-09-17 18:56               ` Jon Steinhart
  1 sibling, 1 reply; 52+ messages in thread
From: Bakul Shah @ 2021-09-17 18:34 UTC (permalink / raw)
  To: TUHS main list

On Sep 17, 2021, at 10:51 AM, Jon Steinhart <jon@fourwinds.com> wrote:
> 
> Bakul Shah writes:
>> I have mixed feelings about this. Unix didn't "throw away"
>> the mainframe world of computing. It simply created a new
>> ecosystem, more suited for the microprocessor age. For IBM it
>> was perhaps the classic Innovator's Dilemma. Similarly now we
>> have (mostly) the Linux ecosystem, while the actual hardware
>> has diverged a lot from the C memory model. There are
>> security issues. There is firmware running on these system
>> about which the OS knows nothing. We have processors like
>> Esperanto Tech's 1088 64 bit Risc-V cores, each with its own
>> vector/tensor unit, 160MB onchip sram and 23.8B transistors
>> but can take only limited advantage of it. We have super
>> performant GPUs but programming them is vendor dependent and
>> a pain. If someone can see a clear path through all this,
>> and create a new software system, they will simply generate a
>> new ecosystem and not worry about 50 years worth of work.
> 
> You're kind of reminding me of the HEP (heterogeneous element
> processor) talk that I saw at I think Usenix in Santa Monica.
> My opinion is that it was a "kitchen sink" project - let's
> throw in a few of these and a few of those and so on.  Also
> analogous to what I saw in the housing market up here when
> people started cashing in their California huts for Oregon
> mansions - when we lived in California we could afford two
> columns out front but now we can afford 6 columns, 8 poticos,
> 6 dormers, 4 turrets, and so on.  Just because you can built
> it doesn't keep it from being an ugly mess.
> 
> So my question on many of these processors is, has anybody
> given any thought to system architecture?  Most likely all
> of us have had to suffer with some piece of spiffy hardware
> that was pretty much unprogrammable.  Do the performance
> numbers mean anything if they can't be achieved in an actual
> system configuration?

If you look at the chip architecture, it is pretty regular.
    https://www.esperanto.ai/technology/
and very low power (0.01W/core as opposed to 7W/core on X86-64)
     https://www.servethehome.com/esperanto-et-soc-1-1092-risc-v-ai-accelerator-solution-at-hot-chips-33/

The equivalent of turrets and porticos and columns and dormers
are IO ports like USB, ethernet, and various GPIOs etc. but they
use only a small portion of the available gates.

IMHO the real issue is that the software folks are *not* providing
and *can no*t provide any sort of guidance for general purpose
computing, as the memory underlying modern programming languages
is so far removed from reality. The situation is sorta like what
happens with people with newly acquired incredible wealth but no
background in how to spend or manage it wisely (and I don't mean
*investing* to get more wealth). At least in that case there are
people who can help you and a tremendous need. Here we can put
billions and billions of gates on a chip and even do wafer scale
integration but these gates are not fungible like money.


^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 18:34             ` Bakul Shah
@ 2021-09-17 18:56               ` Jon Steinhart
  2021-09-17 19:16                 ` Bakul Shah
  0 siblings, 1 reply; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17 18:56 UTC (permalink / raw)
  To: TUHS main list

Bakul Shah writes:
>
> IMHO the real issue is that the software folks are *not* providing
> and *can no*t provide any sort of guidance for general purpose
> computing, as the memory underlying modern programming languages
> is so far removed from reality. The situation is sorta like what
> happens with people with newly acquired incredible wealth but no
> background in how to spend or manage it wisely (and I don't mean
> *investing* to get more wealth). At least in that case there are
> people who can help you and a tremendous need. Here we can put
> billions and billions of gates on a chip and even do wafer scale
> integration but these gates are not fungible like money.

Are you sure about this?  Or is it just that the hardware folks got to
work without ever asking software/systems people for input?

Reminds me of a UNIX workstation project that I was working on at a startup
in the mid-1980s.  The graphics hardware guy came from DEC and had his
designs well under way before I was hired.  I looked at his specs and
asked WTF?  He had put a lot of work, for example, into blazingly fast
frame-buffer clears.  I asked him why and he told me that that was a very
important thing to DECs customers.  He was crestfallen when I told him
that I would never use that hunk of hardware because we were building a
windowing system and not a standalone graphics terminal.  So yeah, he
built useless hardware because he never asked for system design help.

It's not just compute hardware.  Was working on a project in the 1990s
that you might be connected to if you're ever in the emergency room.
After I hired on, I looked over the schematics for the device and kept
asking "What's this for?"  Answer was consistently "Thought that it
would be useful for you software folks."  My response was "Hey, we're
trying to build a low-cost device here.  We don't need it, take it out."

The common thread here, which I don't know if it applies to your example,
is that hardware has traditionally had a longer lead time than software
so commonly a hardware team is assembled and busy designing long before
any software/systems types are brought on board.

Sometimes it makes sense.  I'll never forget a day that I expected to be
walked to the door at a company but to my surprise management backed me
up.  A big failing of the company was that the lead chip designer was
also a founder and on the board so was hard to hold to account.  This
person could always find a better way to do something that had already
been done, so perpetual chip redesigns meant that the chip never happened.
I was in a weekly status meeting where the hardware team presented its
status and schedule.  When it was question time, I asked something like
"So, if I look at your schedule this has 2 weeks to go and this has 5 and
so on, meaning that there's 20 weeks of work left.  Does that mean that
we're not gonna tape out next week like it says on the schedule?"  I got
reamed up and down for not showing respect for the ever so hard working
hardware team.  I think that what saved my butt was pointing out that I
was not criticizing the hardware team, but that we had a big hiring plan
for the software team once the hardware was on its way, and only so much
money in the bank, and maybe we shouldn't be spending that money quite
yet.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 18:56               ` Jon Steinhart
@ 2021-09-17 19:16                 ` Bakul Shah
  2021-09-17 19:35                   ` Jon Steinhart
  0 siblings, 1 reply; 52+ messages in thread
From: Bakul Shah @ 2021-09-17 19:16 UTC (permalink / raw)
  To: Jon Steinhart; +Cc: TUHS main list

On Sep 17, 2021, at 11:56 AM, Jon Steinhart <jon@fourwinds.com> wrote:
> 
> Bakul Shah writes:
>> 
>> IMHO the real issue is that the software folks are *not* providing
>> and *can no*t provide any sort of guidance for general purpose
>> computing, as the memory underlying modern programming languages
>> is so far removed from reality. The situation is sorta like what
>> happens with people with newly acquired incredible wealth but no
>> background in how to spend or manage it wisely (and I don't mean
>> *investing* to get more wealth). At least in that case there are
>> people who can help you and a tremendous need. Here we can put
>> billions and billions of gates on a chip and even do wafer scale
>> integration but these gates are not fungible like money.
> 
> Are you sure about this?  Or is it just that the hardware folks got to
> work without ever asking software/systems people for input?

Let me ask you.

You have a budget of 24 Billion transistors (& a much more limited
power budget). How would you spend it to maximize general purpose
computing?

Note that Esperanto Tech. was founded by Dave Ditzel, not exactly a
h/w newbie. [EE/CS, worked with Dave Patterson, CTO @ Sun, Transmeta
founder etc. etc.]. Now it is entirely possible that Esperanto folks
are building such processors for special purpose applications like
Machine learning but the fact is we simply have an incredible riches
of transistors that we don't how to spend wisely, [My opinion, of course]

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 19:16                 ` Bakul Shah
@ 2021-09-17 19:35                   ` Jon Steinhart
  0 siblings, 0 replies; 52+ messages in thread
From: Jon Steinhart @ 2021-09-17 19:35 UTC (permalink / raw)
  To: TUHS main list

Bakul Shah writes:
> On Sep 17, 2021, at 11:56 AM, Jon Steinhart <jon@fourwinds.com> wrote:
> > 
> > Bakul Shah writes:
> >> 
> >> IMHO the real issue is that the software folks are *not* providing
> >> and *can no*t provide any sort of guidance for general purpose
> >> computing, as the memory underlying modern programming languages
> >> is so far removed from reality. The situation is sorta like what
> >> happens with people with newly acquired incredible wealth but no
> >> background in how to spend or manage it wisely (and I don't mean
> >> *investing* to get more wealth). At least in that case there are
> >> people who can help you and a tremendous need. Here we can put
> >> billions and billions of gates on a chip and even do wafer scale
> >> integration but these gates are not fungible like money.
> > 
> > Are you sure about this?  Or is it just that the hardware folks got to
> > work without ever asking software/systems people for input?
>
> Let me ask you.
>
> You have a budget of 24 Billion transistors (& a much more limited
> power budget). How would you spend it to maximize general purpose
> computing?
>
> Note that Esperanto Tech. was founded by Dave Ditzel, not exactly a
> h/w newbie. [EE/CS, worked with Dave Patterson, CTO @ Sun, Transmeta
> founder etc. etc.]. Now it is entirely possible that Esperanto folks
> are building such processors for special purpose applications like
> Machine learning but the fact is we simply have an incredible riches
> of transistors that we don't how to spend wisely, [My opinion, of course]

Tough question to answer.  I would say that maybe we're getting to the
point where a dream of mine could happen, which is eliminating the
traditional multi-tasking operating system and just having one processor
per process.  One could even call it "hardware containers".  Gak!  Big
unsolved problem the incompatibility between the processes for this sort
of stuff and DRAM processes.  Had hoped that some of the 3D stuff that I
saw Dave D. talk about a few years ago would have panned out by now.

So just because I've never heard anybody else say this, and because we
have named laws like Cole's law, I have Steinhart's law.  It states that
a bad investment to give money to people who have had successful startup
companies.  My opinion is that people are lucky to get it right once, and
very few get it right more than once, so I wouldn't bet on a company
founded by people who have already done successful companies.  A corollary
is that to find a successful startup, look for someone who has been
perfecting an idea over a handful of failed startup companies.  In many
respects, Apple fits that model; they let other people invent stuff and
then they "do it right".

The more interesting question that you raise is, why would you expect
better stuff because we can now do tens of billions of transistors?
From the software side, we have a mind-boggling about of memory, at
least for those of us who began with 4K or less, and I can't see that
it's used wisely.  I've had people tell me that I was wasting my time
being efficient because memory was cheaper than my time.  Which I guess
was true until you multiplied it by the number of systems.  As an EE/CS
guy, I don't really expect the HW people to be any different than the
SW people now that designing HW is just writing SW.

Jon

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 18:07             ` Larry McVoy
@ 2021-09-17 21:03               ` Derek Fawcus
  2021-09-17 22:11                 ` Larry McVoy
  0 siblings, 1 reply; 52+ messages in thread
From: Derek Fawcus @ 2021-09-17 21:03 UTC (permalink / raw)
  To: TUHS main list

On Fri, Sep 17, 2021 at 11:07:10AM -0700, Larry McVoy wrote:
> 
> Apple, much to my surprise, has been thinking about this stuff.
> The M1 is one of the better thought out chips in a long time.

> I especially liked the idea of slow (and low battery draw)
> processors for some apps, and fast ones for apps that need
> fast.  It's an obvious idea in retrospect but I hadn't thought
> to do it that way.  Clever.

I'd suggest that is not Apple, but ARM.  That was sort of the whole
point of their BIG.little architecture with performance and efficiency
cores.

DF

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 21:03               ` Derek Fawcus
@ 2021-09-17 22:11                 ` Larry McVoy
  2021-09-19  4:05                   ` Theodore Ts'o
  0 siblings, 1 reply; 52+ messages in thread
From: Larry McVoy @ 2021-09-17 22:11 UTC (permalink / raw)
  To: Derek Fawcus; +Cc: TUHS main list

On Fri, Sep 17, 2021 at 10:03:42PM +0100, Derek Fawcus wrote:
> On Fri, Sep 17, 2021 at 11:07:10AM -0700, Larry McVoy wrote:
> > 
> > Apple, much to my surprise, has been thinking about this stuff.
> > The M1 is one of the better thought out chips in a long time.
> 
> > I especially liked the idea of slow (and low battery draw)
> > processors for some apps, and fast ones for apps that need
> > fast.  It's an obvious idea in retrospect but I hadn't thought
> > to do it that way.  Clever.
> 
> I'd suggest that is not Apple, but ARM.  That was sort of the whole
> point of their BIG.little architecture with performance and efficiency
> cores.

Credit to ARM, but M1 was where I saw it first.  The M1 performance is
pretty impressive as well.  Reminds me of LED bulbs: "Wait, these are
brighter, use less power, and cost less?  How is that a thing?"

^ permalink raw reply	[flat|nested] 52+ messages in thread

* Re: [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe)
  2021-09-17 22:11                 ` Larry McVoy
@ 2021-09-19  4:05                   ` Theodore Ts'o
  0 siblings, 0 replies; 52+ messages in thread
From: Theodore Ts'o @ 2021-09-19  4:05 UTC (permalink / raw)
  To: Larry McVoy; +Cc: TUHS main list

On Fri, Sep 17, 2021 at 03:11:49PM -0700, Larry McVoy wrote:
> > I'd suggest that is not Apple, but ARM.  That was sort of the whole
> > point of their BIG.little architecture with performance and efficiency
> > cores.
> 
> Credit to ARM, but M1 was where I saw it first.  The M1 performance is
> pretty impressive as well.  Reminds me of LED bulbs: "Wait, these are
> brighter, use less power, and cost less?  How is that a thing?"

ARM's BIG.little architecture dates back to 2011.  Of course, they
didn't *tell* anyone that they were doing this when before the chip
was released, lest it get copied by their competitors.  So they
released a hacked-up version of the Linux kernel that supported their
chip.  And successive versions of BIG.little implemented by different
ARM vendors had their own vendor-specific tweaks, with "board support
kernels" that were heavily hacked up Linux kernels in code that was
never sent upstream, since by the time vendors had released that chip,
their kernel team was moved to working on a new project, where they
would fork the current kernel version, and add completely different
hacks for the next year's version of their System on a Chip (SOC).

Proper support for differntly sized/powered cores didn't land in theb
ppstream Linux until 2019, when Linux 5.0 finally acquired support for
"Energy Aware Scheduling".

So not only does hardware engineering take longer, but it's done in
Sekrit, by software teams employed by chip manufacturers who often
don't have any long-time attachment to the OS, and who will get
reassigned to work the next year's SOC as soon as the current year's
SOC is released.  :-(

Things have gotten a bit better in the last five years, but that's a
pretty low bar, considering how horrible things were before that!

						- Ted

^ permalink raw reply	[flat|nested] 52+ messages in thread

end of thread, other threads:[~2021-09-19  4:06 UTC | newest]

Thread overview: 52+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-01 21:58 [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
2021-09-02  8:42 ` Tony Finch
2021-09-03  0:19   ` John Cowan
2021-09-03  3:24     ` Douglas McIlroy
2021-09-03 13:21       ` Theodore Ts'o
2021-09-08 11:14         ` Tony Finch
2021-09-16 19:27         ` Dan Cross
2021-09-17  0:34           ` Theodore Ts'o
2021-09-17  0:44             ` Larry McVoy
2021-09-17 17:07               ` Bakul Shah
2021-09-17  1:33             ` Dan Cross
2021-09-02 15:41 ` Kevin Bowling
2021-09-02 20:12   ` Marshall Conover
2021-09-03 15:56 ` Warner Losh
2021-09-03 17:10   ` Adam Thornton
2021-09-03 17:28     ` Larry McVoy
2021-09-03 17:42       ` John Floren
2021-09-03 19:02       ` Lawrence Stewart
2021-09-03 19:11       ` Clem Cole
2021-09-03 17:46     ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware [ really a comment on SoCs ] Jon Steinhart
2021-09-16 18:38 ` [TUHS] ATC/OSDI'21 joint keynote: It's Time for Operating Systems to Rediscover Hardware (Timothy Roscoe) Dan Cross
2021-09-16 19:34   ` Jon Steinhart
2021-09-16 19:41     ` Larry McVoy
2021-09-16 23:14       ` Marshall Conover
2021-09-16 23:44         ` Rob Pike
2021-09-17  0:37           ` Larry McVoy
2021-09-17  1:38         ` Jon Steinhart
2021-09-17  3:54         ` John Cowan
2021-09-16 23:45       ` Jon Steinhart
2021-09-17  0:06         ` Al Kossow
2021-09-17  4:06           ` John Cowan
2021-09-17  4:18             ` Al Kossow
2021-09-17  0:32         ` Larry McVoy
2021-09-16 23:54       ` David Arnold
2021-09-17  1:10         ` Jon Steinhart
2021-09-17  1:28           ` Larry McVoy
2021-09-17  1:40             ` Jon Steinhart
2021-09-17  2:04               ` Larry McVoy
2021-09-17  2:21                 ` Jon Steinhart
2021-09-17  2:48           ` Theodore Ts'o
2021-09-17 17:39         ` Bakul Shah
2021-09-17 17:51           ` Jon Steinhart
2021-09-17 18:07             ` Larry McVoy
2021-09-17 21:03               ` Derek Fawcus
2021-09-17 22:11                 ` Larry McVoy
2021-09-19  4:05                   ` Theodore Ts'o
2021-09-17 18:34             ` Bakul Shah
2021-09-17 18:56               ` Jon Steinhart
2021-09-17 19:16                 ` Bakul Shah
2021-09-17 19:35                   ` Jon Steinhart
2021-09-17 15:56     ` Bakul Shah
2021-09-17 18:24       ` ron minnich

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).