Re: [TUHS] SunOS code?

Re: [TUHS] SunOS code?

[Noel Chiappa]

    > From: Warner Losh

    > The trouble, as I was given to understand when I worked at Solbourne,
    > was that ... There were a number of third party bits and pieces in there
    > that could not be relicensed ... if there are other IP issues, not
    > limited ... It's that quagmire that efforts like this will run up
    > against.

Oh, we'll just ask Oracle for a license 'for all parts of SunOS for which they
have the ability to grant a licence'.

There's no way I'd want them to have to chase down all the corner cases,
that's just a way to guarantee it will never happen. I'd want to ask for the
bare minimum of time/effort on their part.

Anything above that, probably the SCCS stuff would be next on the priority
list, sounds like.

	Noel

Re: [TUHS] SunOS code?

[Kevin Bowling]

The effects of copyright on abandonware have been discussed pretty
widely in other communities.  The primary issue is the contracts and
right don't simply expire and it's rare for a company to completely
shutter (i.e. assets including these copyrights are acquired or given
to creditors).

Oracle would need to establish that they have providence over the
files and code, and that exceptions for Novel and other code were
covered by the OpenSolaris rights reviews.  I imagine it might cost
low hundreds of thousands of dollars to do lawyers being what they are
and commercial *nix being such a melting pot of sources.  Since there
is no conceivable revenue stream and Oracle isn't much for goodwill I
am highly skeptical.

Sun releasing OpenSolaris when they finally did and under the CDDL was
pretty tone deaf to what was going on in the market with Linux, but
you have to admire the amount of contract review and legal work that
must have taken.

Regards,
Kevin


On Thu, Aug 30, 2018 at 2:34 PM, Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:
>     > From: Warner Losh
>
>     > The trouble, as I was given to understand when I worked at Solbourne,
>     > was that ... There were a number of third party bits and pieces in there
>     > that could not be relicensed ... if there are other IP issues, not
>     > limited ... It's that quagmire that efforts like this will run up
>     > against.
>
> Oh, we'll just ask Oracle for a license 'for all parts of SunOS for which they
> have the ability to grant a licence'.
>
> There's no way I'd want them to have to chase down all the corner cases,
> that's just a way to guarantee it will never happen. I'd want to ask for the
> bare minimum of time/effort on their part.
>
> Anything above that, probably the SCCS stuff would be next on the priority
> list, sounds like.
>
>         Noel

Re: [TUHS] SunOS code?

[Cág]

Kevin Bowling wrote:

> Sun releasing OpenSolaris when they finally did and under the CDDL was
> pretty tone deaf to what was going on in the market with Linux, but
> you have to admire the amount of contract review and legal work that
> must have taken.

How is OpenSolaris different from SunOS (Sun/Oracle Solaris) anyway?
Isn't the relationship kinda RHEL-CentOS'ish? I.e. one is community-
-supported, and another is commercially.

--
caóc

Re: [TUHS] SunOS code?

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 472 bytes --]

On Fri, Aug 31, 2018 at 5:36 PM Cág <ca6c@bitmessage.ch> wrote:

> How is OpenSolaris different from SunOS (Sun/Oracle Solaris) anyway?
> Isn't the relationship kinda RHEL-CentOS'ish? I.e. one is
> community--supported, and another is commercially.
>
Yikes -- this is like Rolls Royce cutting a deal with GM and produce a new
car based on the Chevy Sububuran, painting it, adding a few details  and
calling it a Silver Ghost.


SunOS != is not Solaris

ᐧ

[-- Attachment #2: Type: text/html, Size: 1301 bytes --]

Re: [TUHS] SunOS code?

[Arthur Krewat]

[-- Attachment #1: Type: text/plain, Size: 289 bytes --]

On 8/31/2018 5:39 PM, Clem Cole wrote:
> SunOS != is not Solaris
>
> ᐧ

Confusion may arise because the last released SunOS 4.1.4 was called 
"Solaris 1.1.2".

Solaris nor SunOS were ever "community supported" although one could 
make the case that SunOS was more generally liked ;)





[-- Attachment #2: Type: text/html, Size: 1183 bytes --]

Re: [TUHS] SunOS code?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 940 bytes --]

On Fri, Aug 31, 2018 at 3:36 PM Cág <ca6c@bitmessage.ch> wrote:

> Kevin Bowling wrote:
>
> > Sun releasing OpenSolaris when they finally did and under the CDDL was
> > pretty tone deaf to what was going on in the market with Linux, but
> > you have to admire the amount of contract review and legal work that
> > must have taken.
>
> How is OpenSolaris different from SunOS (Sun/Oracle Solaris) anyway?
> Isn't the relationship kinda RHEL-CentOS'ish? I.e. one is community-
> -supported, and another is commercially.
>

No. Not even close.

SunOS is BSD based (first 4.2 then 4.3 then with some small amount of
System V code) with a written from scratch vm.

Solaris is sun's do-over based on System V Release 4. It's great leap
backwards. It was a huge slap in the face of the old SunOS crew by inept
management. The final indignity was SunOS 4.1 being rebranded Solaris 1.0,
which was pure marketing...

Warner

[-- Attachment #2: Type: text/html, Size: 1313 bytes --]

Re: [TUHS] SunOS code?

[Larry McVoy]

On Fri, Aug 31, 2018 at 04:34:51PM -0500, C??g wrote:
> Kevin Bowling wrote:
> 
> > Sun releasing OpenSolaris when they finally did and under the CDDL was
> > pretty tone deaf to what was going on in the market with Linux, but
> > you have to admire the amount of contract review and legal work that
> > must have taken.
> 
> How is OpenSolaris different from SunOS (Sun/Oracle Solaris) anyway?
> Isn't the relationship kinda RHEL-CentOS'ish? I.e. one is community-
> -supported, and another is commercially.

SunOS was based on the BSD code, the entire system was frequently
described as "a bug fixed BSD".  It was, of course, a lot more than that,
Sun dis shared libs, a new (much, much better) VM system, invented the
vnode interface that virtualized file systems, BSD had none of that.

Lots of Usenix papers describing that work here:

SunOS felt very much like BSD, all the stuff you thought would work,
usually did, the user space was very BSD like.

http://mcvoy.com/lm/papers/

Solaris was Sys Vr4 (which, if I recall correctly, differed from r3
largely due to some stuff being ported over from SunOS).  Both the kernel
and user space went to a Sys V compat system, it no longer felt anything
like BSD.

So why would Sun take something that everyone loved and replace it
with that steaming pile of Sys V garbage?  Only the top execs actually
knew the real reason, I'm 99% sure my boss, Ken Okin (VP of all server
hardware), did not know the real reason.  Which was, Sun was in some
financial trouble, I don't know the details, but AT&T bought $200M of
Sun stock at 35% over market to bail them out.  In return, Sun had to
drop SunOS and go with Sys V.  AT&T was betting that Sun would make Sys
V as popular as SunOS.

Didn't happen.  Not even close.  A lot of people just bailed, trying to
work with Sys V was miserable.  It put us backwards at least 10 years and
I'd argue more than that.  The engineers loved SunOS and tons of work
got done on the system that management never asked for, the engineers
really drove the agenda.  When all that work was yanked away, it took
the heart out of engineering.

A bunch of people stuck around and tried, they really did and I applaud
them for it but the damage was done.  I was gone by the time ZFS came out
so I have no idea how that passed through the formal vetting process that
Sun had in place.  When I was there, if I had proposed a file system that
wouldn't use the page cache, you'd have to copy from the buffer cache
into the page cache to get mmap to work, I would have been kicked out
of the room and probably kicked out of the kernel group.  We had spent
SO FRIGGING MUCH TIME getting rid of the buffer cache, precisely because
trying to maintain coherency between the page cache (mmap) and the buffer
cache (read/write), it was clear that you wanted a unified model.

Yet the wise heads in charge approved ZFS.  It boggles my mind.  Yeah,
I get it, it's got lots of nice features.  But at the expense of breaking
one of the cornerstones of the kernel design.

The only thing I can think of is that the people who could see the whole
kernel architecture had left, I can't imagine Kleiman, Gingell, Rosenthal,
Powell, any of the old school distinguished engineers, tolerating that
for 1 second.  So my guess is they were gone and nobody in the vetting
process saw the whole picture any more.

If anyone is lurking on the list and was there for the ZFS work, I'd
love to hear your take on it.  I'm speculating.  It just blows my mind
that something that was one of the main design points for the previous
10-15 years, was ignored.  We're not talking about some obscure device
driver, we're talking about mmap(), which was one of the reasons Sun ran
circles around the other guys, they all had crappy mmap() implementations
that copied.  ZFS went back to that.  Weird.

Re: [TUHS] SunOS code?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

On Fri, Aug 31, 2018 at 3:59 PM Larry McVoy <lm@mcvoy.com> wrote:

> A bunch of people stuck around and tried, they really did and I applaud
> them for it but the damage was done.  I was gone by the time ZFS came out
> so I have no idea how that passed through the formal vetting process that
> Sun had in place.  When I was there, if I had proposed a file system that
> wouldn't use the page cache, you'd have to copy from the buffer cache
> into the page cache to get mmap to work, I would have been kicked out
> of the room and probably kicked out of the kernel group.  We had spent
> SO FRIGGING MUCH TIME getting rid of the buffer cache, precisely because
> trying to maintain coherency between the page cache (mmap) and the buffer
> cache (read/write), it was clear that you wanted a unified model.
>

It's reason #1 why Netflix can't use ZFS: the double copy problem...

Warner

[-- Attachment #2: Type: text/html, Size: 1222 bytes --]

Re: [TUHS] SunOS code?

[Cág]

Larry McVoy wrote:

> SunOS was based on the BSD code, the entire system was frequently
> described as "a bug fixed BSD".
> Solaris was Sys Vr4 [...].  Both the kernel and user space went to a
> Sys V compat system, it no longer felt anything like BSD.
 
I find it kinda weird, considering what Bill Joy did for BSD and,
obviously, for Sun. I wonder what he himself thinks about it. Shout out
to Bill if he reads the list.

Also, I'm using the original vi (ex-vi that is from Heirloom), not nvi,
as my development platform. Another weird thing: his original vi never
made it to any BSD distribution.

--
caóc

Re: [TUHS] SunOS code?

[Cág]

Forgot to add: thanks for the good read, Larry!

--
caóc

Re: [TUHS] SunOS code?

[Jon Forrest]

On 8/31/2018 3:19 PM, Cág wrote:

> Also, I'm using the original vi (ex-vi that is from Heirloom), not nvi,
> as my development platform. Another weird thing: his original vi never
> made it to any BSD distribution.

Are you quite sure? I remember using BSD on a VAX in about 1978
when vi just came out. I'm pretty sure Bill Joy wrote the man
page and other documentation.

Maybe it depends on what you mean by "his original vi".

Jon

Re: [TUHS] SunOS code?

[Cág]

Jon Forrest wrote:

> Are you quite sure? I remember using BSD on a VAX in about 1978
> when vi just came out. I'm pretty sure Bill Joy wrote the man
> page and other documentation.

*Open-source BSD distribution. I think it had some license restrictions
when the Jolitzes made the 386 port, so they had to put an alternative.
Then it was Elvis, nvi was written later on.

--
caóc

Re: [TUHS] SunOS code?

[Jon Forrest]

On 8/31/2018 3:30 PM, Cág wrote:
> Jon Forrest wrote:
> 
>> Are you quite sure? I remember using BSD on a VAX in about 1978
>> when vi just came out. I'm pretty sure Bill Joy wrote the man
>> page and other documentation.
> 
> *Open-source BSD distribution. I think it had some license restrictions
> when the Jolitzes made the 386 port, so they had to put an alternative.
> Then it was Elvis, nvi was written later on.

Those of us who used pre-open-source BSD probably have a different
recollection of what BSD was like than those who came later.

Jon

Re: [TUHS] SunOS code?

[Arthur Krewat]

On 8/31/2018 5:58 PM, Larry McVoy wrote:
>
> Solaris was Sys Vr4 (which, if I recall correctly, differed from r3
> largely due to some stuff being ported over from SunOS).  Both the kernel
> and user space went to a Sys V compat system, it no longer felt anything
> like BSD.
>

I would be very interested in anyone's recollections of how Solaris 
eventually turned out performance-wise, say version 9+, compared to 
other operating systems. SunOS, Linux, AIX, etc.

I find it's about equal, and even exceeds Linux in terms of it's NUMA 
support and multi-processor support. I need to move some systems away 
from Solaris and off to Linux, and I find it's NUMA support lacking in 
certain ways.

ak

Re: [TUHS] SunOS code?

[Larry McVoy]

On Fri, Aug 31, 2018 at 07:02:36PM -0400, Arthur Krewat wrote:
> On 8/31/2018 5:58 PM, Larry McVoy wrote:
> >
> >Solaris was Sys Vr4 (which, if I recall correctly, differed from r3
> >largely due to some stuff being ported over from SunOS).  Both the kernel
> >and user space went to a Sys V compat system, it no longer felt anything
> >like BSD.
> >
> 
> I would be very interested in anyone's recollections of how Solaris
> eventually turned out performance-wise, say version 9+, compared to other
> operating systems. SunOS, Linux, AIX, etc.

I did some updating of lmbench [*] when I was dancing with Netflix.  I
need to check that in and publish that because the bits have rotted
since 1995.  I have no idea if that will measure what you want, it's
a bunch of microbenchmarks that measure bandwidth and latency of 
everything I could think of: network, disks, file systems, caches,
memory, etc.  It's interesting but might not be so to you.

If you want to know if Solaris is at the same place as Linux, I can
pretty much promise it is not in the "getting out of the way of the
application".  Solaris was known as Slowaris for a reason.  Yeah, I
know it sucked harder in the beginning, but those were order of 
magnitude suckages.  Linus always cared about performance, he had
a big hand in lmbench, we both cared.

Linus was about performance like I am about code size.  I ran the
BitKeeper dev team for almost 20 years.  I loved a changeset that removed
more lines than it added.  We got up to about 120K lines of code in the
core and then we stayed there for the next decade, added so much more
stuff but always found a way to delete stuff so we didn't get bloated.

But all that said, you need to be specific about what perf you care 
about.  These days the kernel is far more complicated, NUMA etc, 
and you might care about parallel make (I doubt it) or you might care
about Oracle or something like that.

--lm

[*] http://mcvoy.com/lm/papers/lmbench-usenix.pdf

Re: [TUHS] SunOS code?

[Theodore Y. Ts'o]

On Fri, Aug 31, 2018 at 06:57:41PM -0700, Larry McVoy wrote:
> 
> But all that said, you need to be specific about what perf you care 
> about.  These days the kernel is far more complicated, NUMA etc, 
> and you might care about parallel make (I doubt it) or you might care
> about Oracle or something like that.

It wouldn't surprise me if Solaris was more scalable for gazillion
dollar SMP machines with a huge number of cores.  That was one of the
reason as I recall why Solaris had a reputation of being slow; being
scalable to big (and much more profitable) servers was considered more
important than the smaller systems that were probably more numerous
(but not nearly as profitable).

					- Ted

Re: [TUHS] SunOS code?

[Donald ODona]

At 31 Aug 2018 22:32:06 +0000 (+00:00) from "Cág" <ca6c@bitmessage.ch>:
> Then it was Elvis, nvi was written later on.

"It was originally derived from the first incarnation of elvis, written by Steve Kirkendall, as noted in the README file included in nvi's sources. "
https://en.wikipedia.org/wiki/Nvi#Credits_and_distribution

Re: [TUHS] SunOS code?

[Kevin Bowling]

On Fri, Aug 31, 2018 at 4:02 PM, Arthur Krewat <krewat@kilonet.net> wrote:
> On 8/31/2018 5:58 PM, Larry McVoy wrote:
>>
>>
>> Solaris was Sys Vr4 (which, if I recall correctly, differed from r3
>> largely due to some stuff being ported over from SunOS).  Both the kernel
>> and user space went to a Sys V compat system, it no longer felt anything
>> like BSD.
>>
>
> I would be very interested in anyone's recollections of how Solaris
> eventually turned out performance-wise, say version 9+, compared to other
> operating systems. SunOS, Linux, AIX, etc.

Linux started pulling away fast even on high end systems by the early
2000s.  IBM and SGI dumped a ton of money, knowledge, and talent into
this.  By Linux kernel 2.6 the race was entirely won.

After this HP-UX, AIX, and Solaris persist mainly in mainframe-like
vertical stacks used mainly to host mission critical applications that
are sold in bundles or "solutions"

> I find it's about equal, and even exceeds Linux in terms of it's NUMA
> support and multi-processor support. I need to move some systems away from
> Solaris and off to Linux, and I find it's NUMA support lacking in certain
> ways.

This is pure fantasy.  To understand Linux performance on high core
count and multi-socket machines is to have at least passing knowledge
of Paul McKenney's genius work on RCU [1] and NUMA [2] at Sequent [3]
and on Linux.  IBM bought Sequent, made a favorable patent grant of
RCU for Linux, and the rest history.

There is a single feature I have seen in Solaris NUMA that should be
implemented elsewhere.  It does a micro-benchmark on boot to figure
out the inter-core latency map.  On stupid technology like Intel's
ACPI and Xeon Cluster-on-Die and Sub-NUMA-Clustering, you get bogus
data back in the SRAT table describing where the cores are on the on
chip network it just chops things in half and doesn't reflect where
the cores actually were fused off for yield or binning reasons which
is statistically almost always asymmetric.  On better engineered
technology like IBM's POWER8/9 and OPAL firmware, you get the real
locality information of where cores and cache groups actually are.
Solaris' neat little micro-benchmark would work optimally even on the
brain damaged data from Intel.

[1] http://www2.rdrop.com/~paulmck/RCU/
[2] http://www2.rdrop.com/~paulmck/scalability/
[3] http://www2.rdrop.com/~paulmck/techreports/stingcacm3.1999.08.04a.pdf

Regards,
Kevin

Re: [TUHS] SunOS code?

[Kevin Bowling]

I am surprised how good Sun's technical marketing was for you to think
this.  Linux has scaled better since the early 2000s.  The Solaris
x86-64 port has some real gaffes in it to this day at least as visible
in the OpenSolaris derivative codebases, serialization in the locking
primitives kind of things.

On Fri, Aug 31, 2018 at 8:23 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Fri, Aug 31, 2018 at 06:57:41PM -0700, Larry McVoy wrote:
>>
>> But all that said, you need to be specific about what perf you care
>> about.  These days the kernel is far more complicated, NUMA etc,
>> and you might care about parallel make (I doubt it) or you might care
>> about Oracle or something like that.
>
> It wouldn't surprise me if Solaris was more scalable for gazillion
> dollar SMP machines with a huge number of cores.  That was one of the
> reason as I recall why Solaris had a reputation of being slow; being
> scalable to big (and much more profitable) servers was considered more
> important than the smaller systems that were probably more numerous
> (but not nearly as profitable).
>
>                                         - Ted

Re: [TUHS] SunOS code?

[Larry McVoy]

On Sat, Sep 01, 2018 at 09:29:31AM -0700, Kevin Bowling wrote:
> I am surprised how good Sun's technical marketing was for you to think
> this.  Linux has scaled better since the early 2000s.  The Solaris
> x86-64 port has some real gaffes in it to this day at least as visible
> in the OpenSolaris derivative codebases, serialization in the locking
> primitives kind of things.

I think the SPARC bias was very apparent.  Sun loved their own chips, to
their detriment IMO.  I have no personal knowledge of the x86_64 efforts,
but I do know about the Sun i386 efforts.  That was very looked down
upon by the powers that were, the main kernel group, etc.  Those poor
guys had an uphill battle to get anything integrated, nobody wanted it.

So it would not surprise me in the slightest if the x86_64 was a half
assed effort without a lot of attention to stuff like performance.
I don't think they wanted Solaris/x86 to be a success, they wanted
SPARC to be a success.

It was that kind of attitude that has pissed me off at every company
I have ever worked for.  I'm fine with marketing to customers but I
hate it when people believe their own marketing.  Internally, I think
you should be very skeptical, judgemental, critical, whatever you want
to call it, if your code sucks or your hardware sucks, don't pretend
it doesn't, own it and fix it.  That's how you win.

> On Fri, Aug 31, 2018 at 8:23 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> > On Fri, Aug 31, 2018 at 06:57:41PM -0700, Larry McVoy wrote:
> >>
> >> But all that said, you need to be specific about what perf you care
> >> about.  These days the kernel is far more complicated, NUMA etc,
> >> and you might care about parallel make (I doubt it) or you might care
> >> about Oracle or something like that.
> >
> > It wouldn't surprise me if Solaris was more scalable for gazillion
> > dollar SMP machines with a huge number of cores.  That was one of the
> > reason as I recall why Solaris had a reputation of being slow; being
> > scalable to big (and much more profitable) servers was considered more
> > important than the smaller systems that were probably more numerous
> > (but not nearly as profitable).
> >
> >                                         - Ted

-- 
---
Larry McVoy            	     lm at mcvoy.com             http://www.mcvoy.com/lm 

Re: [TUHS] SunOS code?

[Arthur Krewat]

[-- Attachment #1: Type: text/plain, Size: 962 bytes --]

On 9/1/2018 12:27 PM, Kevin Bowling wrote:
>> I find it's about equal, and even exceeds Linux in terms of it's NUMA
>> support and multi-processor support. I need to move some systems away from
>> Solaris and off to Linux, and I find it's NUMA support lacking in certain
>> ways.
> This is pure fantasy.  To understand Linux performance on high core
> count and multi-socket machines is to have at least passing knowledge
> of Paul McKenney's genius work on RCU [1] and NUMA [2] at Sequent [3]
> and on Linux.  IBM bought Sequent, made a favorable patent grant of
> RCU for Linux, and the rest history.

Thanks :) - I'm basing this on Oracle database performance, for the most 
part, and it's weird way of supporting NUMA on Linux in a bass-ackwards 
sort of way. Nothing I see in the latest RedHat/CentOS tells me it even 
cares about NUMA, but maybe that's more of their "we know better than 
you" mentality and it's all hidden under the covers somewhere.

ak

[-- Attachment #2: Type: text/html, Size: 1407 bytes --]

Re: [TUHS] SunOS code?

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

below...

On Sat, Sep 1, 2018 at 12:35 PM Larry McVoy <lm@mcvoy.com> wrote:

>
> It was that kind of attitude that has pissed me off at every company
> I have ever worked for.  I'm fine with marketing to customers but I
> hate it when people believe their own marketing.  Internally, I think
> you should be very skeptical, judgemental, critical, whatever you want
> to call it, if your code sucks or your hardware sucks, don't pretend
> it doesn't, own it and fix it.  That's how you win.
>

Amen....
ᐧ

[-- Attachment #2: Type: text/html, Size: 1245 bytes --]

Re: [TUHS] SunOS code?

[Theodore Y. Ts'o]

On Sat, Sep 01, 2018 at 01:17:59PM -0400, Arthur Krewat wrote:
> On 9/1/2018 12:27 PM, Kevin Bowling wrote:
> > > I find it's about equal, and even exceeds Linux in terms of it's NUMA
> > > support and multi-processor support. I need to move some systems away from
> > > Solaris and off to Linux, and I find it's NUMA support lacking in certain
> > > ways.
> > This is pure fantasy.  To understand Linux performance on high core
> > count and multi-socket machines is to have at least passing knowledge
> > of Paul McKenney's genius work on RCU [1] and NUMA [2] at Sequent [3]
> > and on Linux.  IBM bought Sequent, made a favorable patent grant of
> > RCU for Linux, and the rest history.
> 
> Thanks :) - I'm basing this on Oracle database performance, for the most
> part, and it's weird way of supporting NUMA on Linux in a bass-ackwards sort
> of way. Nothing I see in the latest RedHat/CentOS tells me it even cares
> about NUMA, but maybe that's more of their "we know better than you"
> mentality and it's all hidden under the covers somewhere.

It wouldn't surprise me if Linux's NUMA performance is pretty weak
compared to Solaris.  There was an attempt to try to make NUMA work
well on Linux, with a lot of the effort coming from IBM and SGI, but
that effort was overtaken by events.  Back in Sequent's day, the
remote to local memory latency was ten to one, so making the system
NUMA aware was critical.  But by early 2000's, the remote to local
ratio was under 3:1 (or 2:1) for 4 socket systems, and with AMD's
"Sufficiently Uniform Memory Organization" (SUMO), the ratio was under
1.5:1 or less.

The main reason for this was that Windows was (and as far as I know,
still is) NUMA oblivious.  So x86 chip and motherboard designers
solved the problem, by brute foruce, in hardware.  So by 2003 or 2004,
the Linux Scalability Effort had more or less petered out.  (You can
see the leftover remnants at http://lse.sourceforge.net)

Fundamentally, the economics of 4 socket and higher machines was such
that for many workloads, scale out was much cheaper than scale up.  So
why buy super-expensive IBM X440, x450, and x460 servers, which were
huge cabinets connected by one or more "scalability cables" (sometimes
referred to as the "scalability bottleneck"), when most of the time,
you could just buy a rack of 2U x86 servers which would be much, much
cheaper?

There were certainly workloads this wasn't applicable, of course.  But
when Sun was selling Sun 10k's to web startups during the dot com
boom, and they were using it to serve web traffic, they probably had
too much VC money to burn, because that was *not* the most cost
effective way to do things.

Don't get me wrong; the Read Copy Update (RCU) technique was certainly
very important, and is responsible for much of Linux's SMP scalability
today.  But these days, when you can get up to 28 cores (56 threads)
on a single socket, the need for more than 2 socket systems is already
somewhat niche, and by the time you get to more than 4 sockets, it's
positively microscopic.  As a result, NUMA support on Linux is
certainly not as strong as it could be, and it wouldn't surprise me
that Solaris has developed much better ways of handling the behemoths
such as Sun Enterprise 10k.

					- Ted

P.S.  IBM made the RCU patent available for any GPL code, well before
Sun decided on the CDDL for Solaris.  So if Sun management had chosen
GPL, they could have used RCU....

Re: [TUHS] SunOS code?

[Kevin Bowling]

On Sat, Sep 1, 2018 at 3:19 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Sat, Sep 01, 2018 at 01:17:59PM -0400, Arthur Krewat wrote:
>> On 9/1/2018 12:27 PM, Kevin Bowling wrote:
>> > > I find it's about equal, and even exceeds Linux in terms of it's NUMA
>> > > support and multi-processor support. I need to move some systems away from
>> > > Solaris and off to Linux, and I find it's NUMA support lacking in certain
>> > > ways.
>> > This is pure fantasy.  To understand Linux performance on high core
>> > count and multi-socket machines is to have at least passing knowledge
>> > of Paul McKenney's genius work on RCU [1] and NUMA [2] at Sequent [3]
>> > and on Linux.  IBM bought Sequent, made a favorable patent grant of
>> > RCU for Linux, and the rest history.
>>
>> Thanks :) - I'm basing this on Oracle database performance, for the most
>> part, and it's weird way of supporting NUMA on Linux in a bass-ackwards sort
>> of way. Nothing I see in the latest RedHat/CentOS tells me it even cares
>> about NUMA, but maybe that's more of their "we know better than you"
>> mentality and it's all hidden under the covers somewhere.
>
> It wouldn't surprise me if Linux's NUMA performance is pretty weak
> compared to Solaris.  There was an attempt to try to make NUMA work
> well on Linux, with a lot of the effort coming from IBM and SGI, but
> that effort was overtaken by events.  Back in Sequent's day, the
> remote to local memory latency was ten to one, so making the system
> NUMA aware was critical.  But by early 2000's, the remote to local
> ratio was under 3:1 (or 2:1) for 4 socket systems, and with AMD's
> "Sufficiently Uniform Memory Organization" (SUMO), the ratio was under
> 1.5:1 or less.

Sorry this is just bogus about being weak compared to Solaris.  Are
you looking back with rosy glasses or have you scanned the code in the
past couple years?  I have and there is nothing particularly special
about Solaris internals here or elsewhere.  In fact, there are a lot
of pessimization all over the place.  As Larry said, a lot of folks in
the Linux community clearly cared about performance.  Although the
Solaris code is fairly clean It's not clear Sun valued performance at
all.  A stroll through arch/*/include/asm/ was enough to convince me
of Larry's claims.  I'm not a Linux fanboy but credit goes where it's
due.

Solaris has lgroups, which are a clean design but that is the extent
of its NUMA support, one shot at placement and scheduling.  Linux has
a NUMA allocator, aware scheduler, NUMA-optimized spinlocks and
mutexes, various subsystems correctly use the primitives, and can use
cgroups to contain or gang things.  There is a userland policy tool
called numad that tries to add some additional runtime affinity and
movement policy decisions.

I agree that architecturally Linux NUMA is nowhere near where Sequent
and especially SGI was.  And the reasons you cite are valid, Linux
implementation is good for maybe 8-16 sockets of modern core count
with a much tighter off chip network than the big dogs were building.

Keep in mind IBM wants to sell RockHoppers and E980s (4 drawers, 16
sockets, 768 threads) for dedicated Linux use which have similar
north/south and east/west off chip networks.  They have a lot of very
talented people on the firmware, kernel, compilers to make these
things work fast, including Paul.

> The main reason for this was that Windows was (and as far as I know,
> still is) NUMA oblivious.  So x86 chip and motherboard designers
> solved the problem, by brute foruce, in hardware.  So by 2003 or 2004,
> the Linux Scalability Effort had more or less petered out.  (You can
> see the leftover remnants at http://lse.sourceforge.net)

Windows' NUMA support is on par with Solaris insofar as there is
domain aware memory allocator and scheduler hierarchy that takes the
domain (and SMT etc) into account.  What Windows lacks is the finely
tuned concurrency primitives and everything else Linux has done..
which Solaris lacks as well.  I'm not even talking about RCU (or epoch
based reclamation or proxy collection or hazard pointers, at least one
of which is not patent encumbered), I'm just talking about the quality
of primitives like spinlocks and mutex and rwlock.  There are big
tradeoffs to the implementations of these in terms of fairness,
progress guarantees, and thread scalability.  Linux leads the pack by
a long shot in this department.

Where you start going beyond Linux-like NUMA IMO is when you get
Irix-like features of page copying, migration, and multiple advanced
placement policies.

> Fundamentally, the economics of 4 socket and higher machines was such
> that for many workloads, scale out was much cheaper than scale up.  So
> why buy super-expensive IBM X440, x450, and x460 servers, which were
> huge cabinets connected by one or more "scalability cables" (sometimes
> referred to as the "scalability bottleneck"), when most of the time,
> you could just buy a rack of 2U x86 servers which would be much, much
> cheaper?

Agreed, this is why x86 has dominated the server market for a long time.

> There were certainly workloads this wasn't applicable, of course.  But
> when Sun was selling Sun 10k's to web startups during the dot com
> boom, and they were using it to serve web traffic, they probably had
> too much VC money to burn, because that was *not* the most cost
> effective way to do things.

Agreed.  Those big margins must have caused them to take their eye off
what mattered right at the time Linux was getting some momentum from
the big HW vendors.

> Don't get me wrong; the Read Copy Update (RCU) technique was certainly
> very important, and is responsible for much of Linux's SMP scalability
> today.  But these days, when you can get up to 28 cores (56 threads)
> on a single socket, the need for more than 2 socket systems is already
> somewhat niche, and by the time you get to more than 4 sockets, it's
> positively microscopic.  As a result, NUMA support on Linux is
> certainly not as strong as it could be, and it wouldn't surprise me
> that Solaris has developed much better ways of handling the behemoths
> such as Sun Enterprise 10k.

The E10k was only a 64-core machine on a tight backplane compared to
other large systems.  It didn't have any of the pressing needs that
Sequent and SGI did with multi-drawer interconnects to drive
excellence in NUMA.

These are strange times.  Intel's been putting out some real doozy
chips.  The mesh in Skylake is a partial improvement over the dual
rings of Haswell (though they did some goofy things to increase
latency in undesirable ways), and they aren't going to continue to
brute force it like IBM did with their 17 metal layer process node..
many SKUs in Cascade Lake will be a dual die design and cost a
hilarious amount of money.  AMD's EPYC is really bad in this
department too, one EPYC behaves identically to a 4 socket system with
extremely poor inter-die latency [1].  I think POWER9 is universally
better and the high bin chips (22 core, 88 thread, mega cache) are
only around $2500 compared to Skylake's absurd $12,000.  POWER9 is a
single on chip NUMA domain for 24 cores.  Google publicly stated they
are using it for GPU servers, and that all their monorepo is built for
multiple ISAs.  Through the grapevine I've heard gmail is running on
POWER9 as well now.  That is pretty competent, the reason Intel is
sucking so bad is because people allowed themselves such lock in.  A
hyperscaler should be able to change between a couple ISAs as needed
between purchasing cycles.

>                                         - Ted
>
> P.S.  IBM made the RCU patent available for any GPL code, well before
> Sun decided on the CDDL for Solaris.  So if Sun management had chosen
> GPL, they could have used RCU...

True.  There is also at least one unencumbered strategy such as epoch
based reclamation which was known about around that time [2]

[1] https://www.servethehome.com/amd-epyc-infinity-fabric-latency-ddr4-2400-v-2666-a-snapshot/
[2] https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf

Regards,
Kevin

Re: [TUHS] SunOS code?

[Theodore Y. Ts'o]

On Sat, Sep 01, 2018 at 10:05:06PM -0700, Kevin Bowling wrote:
> 
> Sorry this is just bogus about being weak compared to Solaris.  Are
> you looking back with rosy glasses or have you scanned the code in the
> past couple years?  I have and there is nothing particularly special
> about Solaris internals here or elsewhere.

I haven't looked at Solaris code; I had just *assumed* that if they
were selling million dollar E10k's, they would have had NUMA support
at *least* as good as SGI's Irix.  And it would have been an excuse
for their pathetic performance on UP and 2-4 SMP systems.

> Keep in mind IBM wants to sell RockHoppers and E980s (4 drawers, 16
> sockets, 768 threads) for dedicated Linux use which have similar
> north/south and east/west off chip networks.  They have a lot of very
> talented people on the firmware, kernel, compilers to make these
> things work fast, including Paul.
> ...
> Where you start going beyond Linux-like NUMA IMO is when you get
> Irix-like features of page copying, migration, and multiple advanced
> placement policies.

One thing to consider is that IBM really only cared about optimizing
hardware for DB2, Oracle, and Webshpere.  That's one of the reason why
you didn't see much in the way of innovative file system work, ala
ZFS.  There was no business justification for pouring 100+ engineer
years to develop a next-generation file systesm --- and they had
already done that once already for GPFS, a cluster file system.  As
far as local disk file system was concerned, the only real business
value it had was to serve as a program loader for DB2 and Websphere.  :-)

(I'm exagerating a little for effect, but *only* a little.)

So as far as NUMA was concerned, there was almost certainly not have
been much perceived business value in having sophisticated
auto-migration for arbitrary workloads in the kernel.  Something basic
which was good enough for Oracle, DB2, etc., was all that would be
needed.  (And if you needed to hire consultants from IBM Global
Services to mind-meld with the configuration documentation in order to
get the best out of your Rockhopper.... well, shucks, darn.  :-)

At IBM the business people really did make the funding decisions of
what to work on.  ZFS could have never happened at IBM because no one
would have thought that a even a tiny number of IBM's current or
potential customer base would abandon AIX or Linux and switch to
Solaris, or buy Sun hardware instead of IBM hardware --- just for the
sake of ZFS.  And that's how decision-makers at IBM really thought.
(And to be fair to those decision-makers, IBM is still in business as
a free-standing business --- and Sun is not.)

  			     	     	- Ted

Re: [TUHS] SunOS code?

[Kevin Bowling]

On Sun, Sep 2, 2018 at 12:43 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> On Sat, Sep 01, 2018 at 10:05:06PM -0700, Kevin Bowling wrote:
>>
>> Sorry this is just bogus about being weak compared to Solaris.  Are
>> you looking back with rosy glasses or have you scanned the code in the
>> past couple years?  I have and there is nothing particularly special
>> about Solaris internals here or elsewhere.
>
> I haven't looked at Solaris code; I had just *assumed* that if they
> were selling million dollar E10k's, they would have had NUMA support
> at *least* as good as SGI's Irix.  And it would have been an excuse
> for their pathetic performance on UP and 2-4 SMP systems.

One would hope so, but that was the strategy that got them eaten by a
grue.  Another funny anecdote about this aloofness.. Linux on sparc64
uses the Relaxed Memory Order mode that the hardware offers .
Solaris.. Total Store Order.  There are tons of things like this in
the code that blow my mind.  I would have been pissed if I were on the
hardware side of SPARC.

>
>> Keep in mind IBM wants to sell RockHoppers and E980s (4 drawers, 16
>> sockets, 768 threads) for dedicated Linux use which have similar
>> north/south and east/west off chip networks.  They have a lot of very
>> talented people on the firmware, kernel, compilers to make these
>> things work fast, including Paul.
>> ...
>> Where you start going beyond Linux-like NUMA IMO is when you get
>> Irix-like features of page copying, migration, and multiple advanced
>> placement policies.
>
> One thing to consider is that IBM really only cared about optimizing
> hardware for DB2, Oracle, and Webshpere.  That's one of the reason why
> you didn't see much in the way of innovative file system work, ala
> ZFS.  There was no business justification for pouring 100+ engineer
> years to develop a next-generation file systesm --- and they had
> already done that once already for GPFS, a cluster file system.  As
> far as local disk file system was concerned, the only real business
> value it had was to serve as a program loader for DB2 and Websphere.  :-)
>
> (I'm exagerating a little for effect, but *only* a little.)

Hmm, I think they've been pretty earnest at wanting to be 2+ years
ahead of the general market with POWER for as long as I can see, lots
of HPC money has been subsidizing that.  Depends on the workload but
bus and memory bandwidth right now with PCIe Gen4 and NvLink can
really cut down on server sprawl.  I've met with the GM/chief
architect and they see OpenPOWER positioned as a full frontal
competitor to Intel Xeon.  I'm fairly disappointed in my
contemporaries for not recognizing the value of a completely open
source firmware and on chip controller stack; especially after the
recent snafu where Intel changed the microcode license to disallow
benchmarks and claimed it was an accident.

Your statements make sense to me with respect to AIX, as Linux has
been the main effort since the 2000s.  GPFS looks neat, I wish it were
open or at least internals documented well enough to study the
implementation academically.

>
> So as far as NUMA was concerned, there was almost certainly not have
> been much perceived business value in having sophisticated
> auto-migration for arbitrary workloads in the kernel.  Something basic
> which was good enough for Oracle, DB2, etc., was all that would be
> needed.  (And if you needed to hire consultants from IBM Global
> Services to mind-meld with the configuration documentation in order to
> get the best out of your Rockhopper.... well, shucks, darn.  :-)

That's probably the dirty little secret.  It's long been profitable to
carefully plan software interrupt handlers, user threads, and memory
allocation even on pedestrian servers if they are running a fixed
function.  I guess Google's Borg and the new workalikes could do
semi-automagic things with cgroups these days.  There is evidence of
people getting pretty crazy with it when we see things like Intel
cache allocation features.

> At IBM the business people really did make the funding decisions of
> what to work on.  ZFS could have never happened at IBM because no one
> would have thought that a even a tiny number of IBM's current or
> potential customer base would abandon AIX or Linux and switch to
> Solaris, or buy Sun hardware instead of IBM hardware --- just for the
> sake of ZFS.  And that's how decision-makers at IBM really thought.
> (And to be fair to those decision-makers, IBM is still in business as
> a free-standing business --- and Sun is not.)

Agreed, one of these companies is doing pretty well with a fat
dividend yield, that other has basically been dismantled for all but a
couple remaining desirable platform control points like Java and
MySQL.

Many things in tech are happy accidents and a small number of
motivated people at the right place and time.  A Sun engineer admitted
on some video I've seen that the green light was really given for ZFS
because they got stumped by some UFS bugs.. once enough of ZFS was
written to test the end to end checksumming features they found out
some of these heisenbugs were LSI HBA and disk firmware issues :o)

Surveying some of these filesystems.. JFS2 is a decent, nowhere near
the capabilities of ZFS but even today it's not in dire need of
replacement.. I suspect another issue complementary to your point is
the standalone storage business is many $B of revenue.  ESS/DS8000 and
the like are preferred revenue.  IBM and HP were more in the SAN game
than Sun and SGI who let the customers configure systems themselves be
used as storage (Sun was using VxFS for a long time, SGI had some CXFS
things IIRC).  Tru64 had a pretty interesting filesystem on paper,
curious if you ever looked at its design since they open sourced it.

Regards,
Kevin

Re: [TUHS] SunOS code?

[Gilles Gravier]

[-- Attachment #1: Type: text/plain, Size: 6720 bytes --]

This link :
https://vetusware.com/download/SunOS%20Source%20Code%204.1.3/?id=13475
seems to have the right file (registration required, but it's free, use a
disposable email).

Beats my having to find a SCSI adaptor, a QIC-150 drive, and trying to read
my old QIC-150 tape with the source code on it...

Gilles

Le mar. 4 sept. 2018 à 13:48, Kevin Bowling <kevin.bowling@kev009.com> a
écrit :

> On Sun, Sep 2, 2018 at 12:43 PM, Theodore Y. Ts'o <tytso@mit.edu> wrote:
> > On Sat, Sep 01, 2018 at 10:05:06PM -0700, Kevin Bowling wrote:
> >>
> >> Sorry this is just bogus about being weak compared to Solaris.  Are
> >> you looking back with rosy glasses or have you scanned the code in the
> >> past couple years?  I have and there is nothing particularly special
> >> about Solaris internals here or elsewhere.
> >
> > I haven't looked at Solaris code; I had just *assumed* that if they
> > were selling million dollar E10k's, they would have had NUMA support
> > at *least* as good as SGI's Irix.  And it would have been an excuse
> > for their pathetic performance on UP and 2-4 SMP systems.
>
> One would hope so, but that was the strategy that got them eaten by a
> grue.  Another funny anecdote about this aloofness.. Linux on sparc64
> uses the Relaxed Memory Order mode that the hardware offers .
> Solaris.. Total Store Order.  There are tons of things like this in
> the code that blow my mind.  I would have been pissed if I were on the
> hardware side of SPARC.
>
> >
> >> Keep in mind IBM wants to sell RockHoppers and E980s (4 drawers, 16
> >> sockets, 768 threads) for dedicated Linux use which have similar
> >> north/south and east/west off chip networks.  They have a lot of very
> >> talented people on the firmware, kernel, compilers to make these
> >> things work fast, including Paul.
> >> ...
> >> Where you start going beyond Linux-like NUMA IMO is when you get
> >> Irix-like features of page copying, migration, and multiple advanced
> >> placement policies.
> >
> > One thing to consider is that IBM really only cared about optimizing
> > hardware for DB2, Oracle, and Webshpere.  That's one of the reason why
> > you didn't see much in the way of innovative file system work, ala
> > ZFS.  There was no business justification for pouring 100+ engineer
> > years to develop a next-generation file systesm --- and they had
> > already done that once already for GPFS, a cluster file system.  As
> > far as local disk file system was concerned, the only real business
> > value it had was to serve as a program loader for DB2 and Websphere.  :-)
> >
> > (I'm exagerating a little for effect, but *only* a little.)
>
> Hmm, I think they've been pretty earnest at wanting to be 2+ years
> ahead of the general market with POWER for as long as I can see, lots
> of HPC money has been subsidizing that.  Depends on the workload but
> bus and memory bandwidth right now with PCIe Gen4 and NvLink can
> really cut down on server sprawl.  I've met with the GM/chief
> architect and they see OpenPOWER positioned as a full frontal
> competitor to Intel Xeon.  I'm fairly disappointed in my
> contemporaries for not recognizing the value of a completely open
> source firmware and on chip controller stack; especially after the
> recent snafu where Intel changed the microcode license to disallow
> benchmarks and claimed it was an accident.
>
> Your statements make sense to me with respect to AIX, as Linux has
> been the main effort since the 2000s.  GPFS looks neat, I wish it were
> open or at least internals documented well enough to study the
> implementation academically.
>
> >
> > So as far as NUMA was concerned, there was almost certainly not have
> > been much perceived business value in having sophisticated
> > auto-migration for arbitrary workloads in the kernel.  Something basic
> > which was good enough for Oracle, DB2, etc., was all that would be
> > needed.  (And if you needed to hire consultants from IBM Global
> > Services to mind-meld with the configuration documentation in order to
> > get the best out of your Rockhopper.... well, shucks, darn.  :-)
>
> That's probably the dirty little secret.  It's long been profitable to
> carefully plan software interrupt handlers, user threads, and memory
> allocation even on pedestrian servers if they are running a fixed
> function.  I guess Google's Borg and the new workalikes could do
> semi-automagic things with cgroups these days.  There is evidence of
> people getting pretty crazy with it when we see things like Intel
> cache allocation features.
>
> > At IBM the business people really did make the funding decisions of
> > what to work on.  ZFS could have never happened at IBM because no one
> > would have thought that a even a tiny number of IBM's current or
> > potential customer base would abandon AIX or Linux and switch to
> > Solaris, or buy Sun hardware instead of IBM hardware --- just for the
> > sake of ZFS.  And that's how decision-makers at IBM really thought.
> > (And to be fair to those decision-makers, IBM is still in business as
> > a free-standing business --- and Sun is not.)
>
> Agreed, one of these companies is doing pretty well with a fat
> dividend yield, that other has basically been dismantled for all but a
> couple remaining desirable platform control points like Java and
> MySQL.
>
> Many things in tech are happy accidents and a small number of
> motivated people at the right place and time.  A Sun engineer admitted
> on some video I've seen that the green light was really given for ZFS
> because they got stumped by some UFS bugs.. once enough of ZFS was
> written to test the end to end checksumming features they found out
> some of these heisenbugs were LSI HBA and disk firmware issues :o)
>
> Surveying some of these filesystems.. JFS2 is a decent, nowhere near
> the capabilities of ZFS but even today it's not in dire need of
> replacement.. I suspect another issue complementary to your point is
> the standalone storage business is many $B of revenue.  ESS/DS8000 and
> the like are preferred revenue.  IBM and HP were more in the SAN game
> than Sun and SGI who let the customers configure systems themselves be
> used as storage (Sun was using VxFS for a long time, SGI had some CXFS
> things IIRC).  Tru64 had a pretty interesting filesystem on paper,
> curious if you ever looked at its design since they open sourced it.
>
> Regards,
> Kevin
>


-- 
*Gilles Gravier*  - Gilles@Gravier.org
GSM : +33618347147 and +41794728437
Skype : ggravier | PGP Key : 0x8DE6D026
<http://pgp.mit.edu:11371/pks/lookup?search=0x8DE6D026&op=index>

[-- Attachment #2: Type: text/html, Size: 8889 bytes --]

Re: [TUHS] SunOS code?

[Henry Bent]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Tue, Sep 4, 2018, 13:40 Gilles Gravier <gilles@gravier.org> wrote:

> This link :
> https://vetusware.com/download/SunOS%20Source%20Code%204.1.3/?id=13475
> seems to have the right file (registration required, but it's free, use a
> disposable email).
>
> Beats my having to find a SCSI adaptor, a QIC-150 drive, and trying to
> read my old QIC-150 tape with the source code on it...
>
> Gilles
>

I feel like we've been through this before on this list, but perhaps it
bears repeating: just because you can find source (or binaries, or CD
images, etc.) on the internet, that doesn't make it the least bit legal.
That is clearly the case here. Sadly, there are even higher profile sites
like the Internet Archive that have this problem too.  The concept of
"abandonware" has no legal footing whatsoever.

-Henry

[-- Attachment #2: Type: text/html, Size: 1420 bytes --]

Re: [TUHS] SunOS code?

[Tony Finch]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]


> On 2 Sep 2018, at 06:05, Kevin Bowling <kevin.bowling@kev009.com> wrote:
> 
> The E10k was only a 64-core machine on a tight backplane compared to
> other large systems.  It didn't have any of the pressing needs that
> Sequent and SGI did with multi-drawer interconnects to drive
> excellence in NUMA.

When I started work at Cambridge in 2002 our central supercomputer was being replaced with a cluster of Sun Fire E15K machines with a fancy interconnect - it topped out at position 199 on the top500 list https://www.top500.org/list/2003/06/?page=2 with a 300 core configuration. It looks like they never managed to get the whole thing working as a single cluster since the other two thirds of the installation had positions 200 and 201! (The Nov. 2002 top500 list has it in 6 x 144 core shards.) Here’s a news item about it: https://www.cnet.com/news/sun-expands-supercomputer-effort/

> True.  There is also at least one unencumbered strategy such as epoch
> based reclamation which was known about around that time [2]
> 
> [2] https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-579.pdf

The big benchmarks in this lovely thesis were run on one of the E15K supercomputer boxes :-)

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at


[-- Attachment #2: Type: text/html, Size: 2256 bytes --]

Re: [TUHS] SunOS code?

[Gilles Gravier]

[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]

(Henri, you're in copy of this again, because first time I did a Reply
instead of Reply to list/all. Sorry.)

Le mar. 4 sept. 2018 à 19:45, Henry Bent <henry.r.bent@gmail.com> a écrit :

>
>
> On Tue, Sep 4, 2018, 13:40 Gilles Gravier <gilles@gravier.org> wrote:
>
>> This link :
>> https://vetusware.com/download/SunOS%20Source%20Code%204.1.3/?id=13475
>> seems to have the right file (registration required, but it's free, use a
>> disposable email).
>>
>> Beats my having to find a SCSI adaptor, a QIC-150 drive, and trying to
>> read my old QIC-150 tape with the source code on it...
>>
>> Gilles
>>
>
> I feel like we've been through this before on this list, but perhaps it
> bears repeating: just because you can find source (or binaries, or CD
> images, etc.) on the internet, that doesn't make it the least bit legal.
> That is clearly the case here. Sadly, there are even higher profile sites
> like the Internet Archive that have this problem too.  The concept of
> "abandonware" has no legal footing whatsoever.
>
> -Henry
>

Oh I definitely know the sources aren't officially accessible. By the way,
I had copies of them (my QIC tape) when I was a student. I still have the
QIC.

It's the common example that I use to tell people that opensourcing
software makes it more secure because the good guys have access to the
source code at the same time as the bad guys, which gives them a fair
chance to fix bugs before the bad guys use them.

With closed source (SunOS, VMS...) the good guys don't have access to the
source code... but the bad guys will always (either by paying somebody
enough to make a copy for them, or by finding them on some non legitimate
place). As a student I had the source of SunOS (4.1.3) but also VMS (on a
few TK-50 tapes).

For me, that vetusware site is certainly not legitimate... but since I have
the QIC tape at home, I just used it as an easy alternative to having to
get the hardware to read my tape back in working order...

I certainly do not consider it a legally approved way of distributing code
which is, as we all agree, NOT open source.

Gilles

-- 
*Gilles Gravier*  - Gilles@Gravier.org
GSM : +33618347147 and +41794728437
Skype : ggravier | PGP Key : 0x8DE6D026
<http://pgp.mit.edu:11371/pks/lookup?search=0x8DE6D026&op=index>

[-- Attachment #2: Type: text/html, Size: 4483 bytes --]

Re: [TUHS] SunOS code?

[Arthur Krewat]

On 9/5/2018 2:31 AM, Gilles Gravier wrote:
> It's the common example that I use to tell people that opensourcing 
> software makes it more secure because the good guys have access to the 
> source code at the same time as the bad guys, which gives them a fair 
> chance to fix bugs before the bad guys use them.


Bash/Shellshock kinda proves that premise incorrect, although it's 
pretty much the worst-case example, but still...  ;)

Announced in 2014, it goes back to September 1989 (according to a 
wikipedia article, so I'm not sure about that date's accuracy).

https://en.wikipedia.org/wiki/Shellshock_(software_bug)

https://www.cvedetails.com/product/47/Linux-Linux-Kernel.html?vendor_id=33
https://www.cvedetails.com/product/17/IBM-AIX.html?vendor_id=14
https://www.cvedetails.com/product/20/HP-Hp-ux.html?vendor_id=10
https://www.cvedetails.com/product/19755/Oracle-Solaris.html?vendor_id=93

It could be argued that the above CVE results are either under-reported 
(closed-source), or over-reported (open-source). Or vice-versa ;)

ak

Re: [TUHS] SunOS code?

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

On Wed, Sep 5, 2018 at 6:55 AM Arthur Krewat <krewat@kilonet.net> wrote:

>
>
> On 9/5/2018 2:31 AM, Gilles Gravier wrote:
> > It's the common example that I use to tell people that opensourcing
> > software makes it more secure because the good guys have access to the
> > source code at the same time as the bad guys, which gives them a fair
> > chance to fix bugs before the bad guys use them.
>
>
> Bash/Shellshock kinda proves that premise incorrect, although it's
> pretty much the worst-case example, but still...  ;)
>

I'm not sure it does. It proves that bugs aren't instantly found, true. It
doesn't provide perfection, but does make it easier to find / fix bugs
before the bad guys. How long would such a bug have languished it if were
buried inside of DCL.B32 instead of being out in the open?

Warner

[-- Attachment #2: Type: text/html, Size: 1181 bytes --]

Re: [TUHS] SunOS code?

[Chet Ramey]

On 9/5/18 11:26 AM, Warner Losh wrote:

>     On 9/5/2018 2:31 AM, Gilles Gravier wrote:
>     > It's the common example that I use to tell people that opensourcing
>     > software makes it more secure because the good guys have access to the
>     > source code at the same time as the bad guys, which gives them a fair
>     > chance to fix bugs before the bad guys use them.
> 
> 
>     Bash/Shellshock kinda proves that premise incorrect, although it's
>     pretty much the worst-case example, but still...  ;)
> 
> 
> I'm not sure it does. It proves that bugs aren't instantly found, true. It
> doesn't provide perfection, but does make it easier to find / fix bugs
> before the bad guys. How long would such a bug have languished it if were
> buried inside of DCL.B32 instead of being out in the open?

It proves that if there is someone who has an idea, or who thinks about a
thing in new ways, he can verify his suspicions without too much trouble.
The barrier to investigation is lowered.

Chet


-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
		 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/

Re: [TUHS] SunOS code?

[Arthur Krewat]

On 9/5/2018 11:26 AM, Warner Losh wrote:
>
> I'm not sure it does. It proves that bugs aren't instantly found, 
> true. It doesn't provide perfection, but does make it easier to find / 
> fix bugs before the bad guys. How long would such a bug have 
> languished it if were buried inside of DCL.B32 instead of being out in 
> the open?

It depends on how it was found in the first place. A quick Google 
doesn't tell me much about exactly how it was discovered initially. Nor 
is there any background information that says it wasn't (or was) 
exploited before the announcement. Was it discovered because someone 
(Stéphane Chazelas) was just reading open source code? Or was he trying 
to do something innocent and it broke in such a way that it was obvious 
bash was doing something bad? Or was he investigating a break-in and 
found the vector? Serious questions, I'd love to hear from anyone who 
knows more.

My original point remains: Open Source doesn't necessarily mean more 
secure if a really bad exploit was allowed to exist for 25 years.

No offense intended to anyone on this list.

ak

Re: [TUHS] SunOS code?

[Dave Horsfall]

On Wed, 5 Sep 2018, Gilles Gravier wrote:

> (Henri, you're in copy of this again, because first time I did a Reply 
> instead of Reply to list/all. Sorry.)

One of my pet hates is people who use "Reply All" because either they are 
too lazy to edit the Reply (and they top-post, too), or the list is set up 
that way.

I really don't need my own personal copy, as well as a list copy.

Oh, and people who are too lazy to trim their replies too, because they
are encouraged to top-post.

-- Dave

[TUHS] Mail etiquette (was: SunOS code?)

[Greg 'groggy' Lehey]

[-- Attachment #1: Type: text/plain, Size: 2026 bytes --]

On Thursday,  6 September 2018 at  9:40:30 +1000, Dave Horsfall wrote:
> On Wed, 5 Sep 2018, Gilles Gravier wrote:
>
>> (Henri, you're in copy of this again, because first time I did a Reply
>> instead of Reply to list/all. Sorry.)
>
> One of my pet hates is people who use "Reply All" because either they are
> too lazy to edit the Reply (and they top-post, too), or the list is set up
> that way.

That's a matter of opinion.  Many people consider it good manners to
specifically mention people participating in a discussion.  For
example,
https://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-questions/article.html#idp53204040
specifically states:

  Unless there is a good reason to do otherwise, reply to the sender
  and to FreeBSD-questions [the list].

> I really don't need my own personal copy, as well as a list copy.

That's what procmail is for.

> Oh, and people who are too lazy to trim their replies too, because
> they are encouraged to top-post.

Yes, that's in that URL too:

  Put your response in the correct place (after the text to which it
  replies). It is very difficult to read a thread of responses where
  each reply comes before the text to which it replies.

Also another thing that I still think very important:

  If the submitter did not abide by format conventions (lines too
  long, inappropriate subject line), please fix it. In the case of an
  incorrect subject line (such as “HELP!!??”), change the subject line
  to (say) “Re: Difficulties with sync PPP (was: HELP!!??)”. That way
  other people trying to follow the thread will have less difficulty
  following it.

That doesn't explicitly address the issue of change of topic within a
thread, but it should.

Greg
--
Sent from my desktop computer.
Finger grog@lemis.com for PGP public key.
See complete headers for address and phone numbers.
This message is digitally signed.  If your Microsoft mail program
reports problems, please read http://lemis.com/broken-MUA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 163 bytes --]