[COFF] Re: [TUHS] the wheel of reincarnation goes sideways

[COFF] Re: [TUHS] the wheel of reincarnation goes sideways

[Dan Cross]

[bumping to COFF]

On Wed, Mar 8, 2023 at 2:05 PM ron minnich <rminnich@gmail.com> wrote:
> The wheel of reincarnation discussion got me to thinking:
>
> What I'm seeing is reversing the rotation of the wheel of reincarnation. Instead of pulling the task (e.g. graphics) from a special purpose device back into the general purpose domain, the general purpose computing domain is pushed into the special purpose device.
>
> I first saw this almost 10 years ago with a WLAN modem chip that ran linux on its 4 core cpu, all of it in a tiny package. It was faster, better, and cheaper than its traditional embedded predecessor -- because the software stack was less dedicated and single-company-created. Take Linux, add some stuff, voila! WLAN modem.
>
> Now I'm seeing it in peripheral devices that have, not one, but several independent SoCs, all running Linux, on one card. There's even been a recent remote code exploit on, ... an LCD panel.
>
> Any of these little devices, with the better part of a 1G flash and a large part of 1G DRAM, dwarfs anything Unix ever ran on. And there are more and more of them, all over the little PCB in a laptop.
>
> The evolution of platforms like laptops to becoming full distributed systems continues.
> The wheel of reincarnation spins counter clockwise -- or sideways?

About a year ago, I ran across an email written a decade or more prior
on some mainframe mailing list where someone wrote something like,
"wow! It just occurred to me that my Athlon machine is faster than the
ES/3090-600J I used in 1989!" Some guy responded angrily, rising to
the wounded honor of IBM, raving about how preposterous this was
because the mainframe could handle a thousand users logged in at one
time and there's no way this Linux box could ever do that.

I was struck by the absurdity of that; it's such a ridiculous
non-comparison. The mainframe had layers of terminal concentrators,
3270 controllers, IO controllers, etc, etc, and a software ecosystem
that made heavy use of all of that, all to keep user interaction _off_
of the actual CPU (I guess freeing that up to run COBOL programs in
batch mode...); it's not as though every time a mainframe user typed
something into a form on their terminal it interrupted the primary
CPU.

Of course, the first guy was right: the AMD machine probably _was_
more capable than a 3090 in terms of CPU performance, RAM and storage
capacity, and raw bandwidth between the CPU and IO subsystems. But the
3090 was really more like a distributed system than the Athlon box
was, with all sorts of offload capabilities. For that matter, a
thousand users probably _could_ telnet into the Athlon system. With
telnet in line mode, it'd probably even be decently responsive.

So often it seems to me like end-user systems are just continuing to
adopt "large system" techniques. Nothing new under the sun.

> I'm no longer sure the whole idea of the wheel or reincarnation is even applicable.

I often feel like the wheel has fallen onto its side, and we're
continually picking it up from the edge and flipping it over, ad
nauseum.

        - Dan C.

[COFF] Re: the wheel of reincarnation goes sideways

[Tom Ivar Helbekkmo via COFF]

Dan Cross <crossd@gmail.com> writes:

> About a year ago, I ran across an email written a decade or more prior
> on some mainframe mailing list where someone wrote something like,
> "wow! It just occurred to me that my Athlon machine is faster than the
> ES/3090-600J I used in 1989!" Some guy responded angrily, rising to
> the wounded honor of IBM, raving about how preposterous this was
> because the mainframe could handle a thousand users logged in at one
> time and there's no way this Linux box could ever do that.
>
> I was struck by the absurdity of that; it's such a ridiculous
> non-comparison.

I did one of those.  Back in the early nineties, I had a 286 box running
MINIX 1.5 as my home workstation, and a similar one running DOS at work.
My job, however, was as one of a team of sysadmins caring for a VAX-780
running VMS.

I used C-TeX to format documents on the DOS PC, and spent a couple of
days porting it to the VMS C compiler.  Performance was utterly dismal
at first, but once I realized that the stdio stuff in the standard
libary was the problem, I modified C-TeX to do output to binary files of
fixed size 512 byte blocks in RMS, the VMS file system.  In the small
hours of the night, I discovered that the big and expensive VAX-780 was
able to pretty much exactly match my 286-box when formatting documents.

The very next day, I found that the same machine did the TeX formatting
just as fast, while a hundred or so other people were actively using it
for their own work.

-tih
-- 
Most people who graduate with CS degrees don't understand the significance
of Lisp.  Lisp is the most important idea in computer science.  --Alan Kay

[COFF] Re: [TUHS] Re: the wheel of reincarnation goes sideways

[John Cowan]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Wed, Mar 8, 2023 at 2:53 PM Dan Cross <crossd@gmail.com> wrote:


> > Now I'm seeing it in peripheral devices that have, not one, but several
> independent SoCs, all running Linux, on one card. There's even been a
> recent remote code exploit on, ... an LCD panel.
>

I remember at one time I had on my desk a PC with an 80x86 CPU and an
Ethernet card that had an 80(x+1)86 chip inside.  I think x=0, but I'm not
sure.


> But the
> 3090 was really more like a distributed system than the Athlon box
> was, with all sorts of offload capabilities. For that matter, a
> thousand users probably _could_ telnet into the Athlon system. With
> telnet in line mode, it'd probably even be decently responsive.
>

I find that difficult to believe.  It seems too high by an order of
magnitude.  Another thing that doesn't get mentioned much is that classic
mainframes had SRAM, so their memory bandwidth was enormous.

[-- Attachment #2: Type: text/html, Size: 1910 bytes --]

[COFF] Re: [TUHS] Re: the wheel of reincarnation goes sideways

[Dan Cross]

On Wed, Mar 8, 2023 at 8:22 PM John Cowan <cowan@ccil.org> wrote:
> On Wed, Mar 8, 2023 at 2:53 PM Dan Cross <crossd@gmail.com> wrote:
>> But the
>> 3090 was really more like a distributed system than the Athlon box
>> was, with all sorts of offload capabilities. For that matter, a
>> thousand users probably _could_ telnet into the Athlon system. With
>> telnet in line mode, it'd probably even be decently responsive.
>
> I find that difficult to believe.  It seems too high by an order of magnitude.

I'm not going to claim it would be zippy, but I do think it would work
acceptably.

Suppose that 1000 users telnet'ed into the x86 machine, but remained
essentially idle; what resources would that consume? We'd have 1000
open TCP connections, a thousand shell processes, a thousand
telnetd's, etc. All of that would consume some amount of RAM (though
there'd be a lot of sharing of text and read-only data and so on),
some VM space requiring RAM for paging structures and so on, some
accounting data in the kernel, 1000 pseudo-ttys allocated, entries in
the process table, etc. But, most of those shells would spend most of
their time blocked waiting on input, so wouldn't consume CPU
continuously, and similarly with the TCP connections mostly idle, the
kernel is not generally wasting a lot of processor time on the login
sessions. There'd be some bookkeeping data on disk, but that would be
small. System overhead would amount to maybe a few megabytes, I'd
imagine.

If all of those users ran telnet in line mode, then the system isn't
getting pounded with interrupts all the time, even if they're
executing commands (the per-character overhead would be absorbed by
the client).

I don't think I have a machine of quite the Athlon vintage, but I _do_
have a machine with a Ryzen processor that's a couple of years old
down in my basement. As an experiment, I wrote a little "expect"
script to login to that machine a thousand times, doing so
recursively: that is, the script starts off ssh'ing into the machine,
and then in that session, logs in again, and so on, a thousand times,
before finally going interactive. I used encryption, public-key
authentication, and compression, and bounced through a "jump host" for
each session, ensuring that I'm using the network for each login. The
effect here is that typing into the final shell sort of simulates 1000
users typing simultaneously, complete with all the glorious interrupt
and scheduler overhead that implies.

Response time in that connection is not bad; certainly on par with the
3090 I used for a while in the early 90s. If I login in another
window, it doesn't even register that there are a thousand "users"
logged in, even if I'm running something chatty in the "thousand
users" window.

By contrast, the mainframe required a tremendous amount of offload
support to shield the CPU from all of that bursty user activity. They
made user actions look like block transfers, thus amortizing (much) of
the overhead of interactivity. With the same load, the mainframe is
storing some state data in memory regarding which users are logged in
or connected or dialed or whatever, but the situation isn't that much
different than mostly-idle telnet connections in line-mode: save that
it's even more favorable to the mainframe in that much of the
interaction is per-screen of data, as opposed to per-line.

The difference in interactivity and offload is why I think the
comparison is poor. If the mainframe handled user sessions the same
way the x86 machine handled telnet logins, I imagine it would be
swamped way worse than the AMD machine (or whatever it was that person
was writing about 10 or 15 years ago). Perhaps a better comparison
would be to a web server that was accepting HTTP requests from 1000
different clients. I'm quite sure that x86 machines of the Athlon era
could cope with that load.

> Another thing that doesn't get mentioned much is that classic mainframes had SRAM, so their memory bandwidth was enormous.

I suspect this has less of a difference than one would hope when
comparing against a modern machine.

The specific comparison in this case was against an IBM 3090-600J. It
appears to use SRAM for cache ("high speed buffer" in IBM-speak), but
seems to use DRAM for central and expanded storage. In this reference
I found on bitsavers, they make a big deal about their "one million
bit memory chip", but that's DRAM
(http://www.bitsavers.org/pdf/ibm/3090/G580-1005-0_The_IBM_3090_Processor_Family_Jul87.pdf;
see "IBM Advances the Technology" on page 10).

Moreover, that machine supported up to 6 CPUs running at a clock rate
of 69 MHz. That same reference says they could bring cycle times down
to 17.2ns using ECL chips; DDR2 can match that. My Mac Studio blows it
out of the water.

For systems older than the 3090, I'm not sure that the SRAM difference
matters much at all: those machines had tiny memories compared to even
modern cell phones, and their CPUs and buses were pitifully slow. Even
if they had more RAM bandwidth than machines now (which I do not think
is really true), they couldn't use it. Indeed, I suspect their total
memory sizes were smaller than L3 cache (which is SRAM) on modern
machines.

        - Dan C.

[COFF] Re: [TUHS] Re: the wheel of reincarnation goes sideways

[Larry McVoy]

On Thu, Mar 09, 2023 at 02:55:44PM -0500, Dan Cross wrote:
> On Wed, Mar 8, 2023 at 8:22???PM John Cowan <cowan@ccil.org> wrote:
> > On Wed, Mar 8, 2023 at 2:53???PM Dan Cross <crossd@gmail.com> wrote:
> >> But the
> >> 3090 was really more like a distributed system than the Athlon box
> >> was, with all sorts of offload capabilities. For that matter, a
> >> thousand users probably _could_ telnet into the Athlon system. With
> >> telnet in line mode, it'd probably even be decently responsive.
> >
> > I find that difficult to believe.  It seems too high by an order of magnitude.
> 
> I'm not going to claim it would be zippy, but I do think it would work
> acceptably.
> 
> Suppose that 1000 users telnet'ed into the x86 machine, but remained
> essentially idle; what resources would that consume? We'd have 1000
> open TCP connections, a thousand shell processes, a thousand
> telnetd's, etc. 

The early Unix code really did not like stuff like this.  Lots of linear
scans through what were assumed to be short lists.  I still remember an
SGI Challenge being brought to it's knees by a bunch of racks of modems.
The same machine could move a ton of data but not when it was being
forced through a zillion sockets.

Linux seems well past that problem but it's possible that back in the
Athlon days it still sucked.  I pinged Linus, if he remembers when the
kernel got taught to scale on sockets I'll report back.

--lm

[COFF] Re: [TUHS] Re: the wheel of reincarnation goes sideways

[Dan Cross]

On Thu, Mar 9, 2023 at 3:09 PM Larry McVoy <lm@mcvoy.com> wrote:
> On Thu, Mar 09, 2023 at 02:55:44PM -0500, Dan Cross wrote:
> > On Wed, Mar 8, 2023 at 8:22???PM John Cowan <cowan@ccil.org> wrote:
> > > On Wed, Mar 8, 2023 at 2:53???PM Dan Cross <crossd@gmail.com> wrote:
> > >> But the
> > >> 3090 was really more like a distributed system than the Athlon box
> > >> was, with all sorts of offload capabilities. For that matter, a
> > >> thousand users probably _could_ telnet into the Athlon system. With
> > >> telnet in line mode, it'd probably even be decently responsive.
> > >
> > > I find that difficult to believe.  It seems too high by an order of magnitude.
> >
> > I'm not going to claim it would be zippy, but I do think it would work
> > acceptably.
> >
> > Suppose that 1000 users telnet'ed into the x86 machine, but remained
> > essentially idle; what resources would that consume? We'd have 1000
> > open TCP connections, a thousand shell processes, a thousand
> > telnetd's, etc.
>
> The early Unix code really did not like stuff like this.  Lots of linear
> scans through what were assumed to be short lists.  I still remember an
> SGI Challenge being brought to it's knees by a bunch of racks of modems.
> The same machine could move a ton of data but not when it was being
> forced through a zillion sockets.

Oh for sure I wouldn't try it on a VAX or PDP-11. I'm a bit surprised
by the SGI thing, to be honest, but only a bit: as you say, I think
that was just before the big push to make Unix really scalable.

> Linux seems well past that problem but it's possible that back in the
> Athlon days it still sucked.  I pinged Linus, if he remembers when the
> kernel got taught to scale on sockets I'll report back.

Thanks, I'm curious what he says.

        - Dan C.

[COFF] Re: [TUHS] Re: the wheel of reincarnation goes sideways

[Bakul Shah]

On Mar 9, 2023, at 11:55 AM, Dan Cross <crossd@gmail.com> wrote:
> 
> Suppose that 1000 users telnet'ed into the x86 machine, but remained
> essentially idle; what resources would that consume? We'd have 1000
> open TCP connections, a thousand shell processes, a thousand
> telnetd's, etc. All of that would consume some amount of RAM (though
> there'd be a lot of sharing of text and read-only data and so on),
> some VM space requiring RAM for paging structures and so on, some
> accounting data in the kernel, 1000 pseudo-ttys allocated, entries in
> the process table, etc. But, most of those shells would spend most of
> their time blocked waiting on input, so wouldn't consume CPU
> continuously, and similarly with the TCP connections mostly idle, the
> kernel is not generally wasting a lot of processor time on the login
> sessions. There'd be some bookkeeping data on disk, but that would be
> small. System overhead would amount to maybe a few megabytes, I'd
> imagine.

Not the same but in 1995 at Real Networks our server s/w running on
a 50Mhz or 100Mhz Pentium could handle 1000 TCP control connections
(mostly idle) and 1000 UDP "streams", each sending 10 packets/second,
which was the limiting factor. IIRC we had reduced per socket tcp
send/recv buffer size to a small number. I don't recall now whether
these machines had more than 16GB but we didn't want to tie up lots
of memory in idle buffers.

We got a real boost in traffic in Oct'95 when people all over the
world wanted to know the verdict in O.J.Simpson's murder trial in
real time! After that I added code for feeding live streams to any
downstream servers so that theoretically a 3 level distribution
tree can deliver live data to a billion people.

[COFF] Re: [TUHS] Re: the wheel of reincarnation goes sideways

[Dan Cross]

I don't know if a thousand users ever logged in there at one time, but
they do tend to have a lot of simultaneous logins.

On Mon, Mar 13, 2023 at 6:16 PM Peter Pentchev <roam@ringlet.net> wrote:
>
> On Wed, Mar 08, 2023 at 02:52:43PM -0500, Dan Cross wrote:
> > [bumping to COFF]
> >
> > On Wed, Mar 8, 2023 at 2:05 PM ron minnich <rminnich@gmail.com> wrote:
> > > The wheel of reincarnation discussion got me to thinking:
> [snip]
> > > The evolution of platforms like laptops to becoming full distributed systems continues.
> > > The wheel of reincarnation spins counter clockwise -- or sideways?
> >
> > About a year ago, I ran across an email written a decade or more prior
> > on some mainframe mailing list where someone wrote something like,
> > "wow! It just occurred to me that my Athlon machine is faster than the
> > ES/3090-600J I used in 1989!" Some guy responded angrily, rising to
> > the wounded honor of IBM, raving about how preposterous this was
> > because the mainframe could handle a thousand users logged in at one
> > time and there's no way this Linux box could ever do that.
> [snip]
> > For that matter, a
> > thousand users probably _could_ telnet into the Athlon system. With
> > telnet in line mode, it'd probably even be decently responsive.
>
> sdf.org (formerly sdf.lonestar.org) comes to mind...
>
> G'luck,
> Peter
>
> --
> Peter Pentchev  roam@ringlet.net roam@debian.org pp@storpool.com
> PGP key:        http://people.FreeBSD.org/~roam/roam.key.asc
> Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13