[TUHS] 8th Edition timeline

[TUHS] 8th Edition timeline

[Paul Ruizendaal]

Here is a question for the old hands from the Labs, I’m trying to get the timeline of some development steps right.

The two main things are: when did the 4.1 merge take place, and when were ‘streams’ added?

Going by file dates, the surviving 8th edition source appears to be from 1985. I can see that a lot of files in /usr/include did not change after Jan 1982 (e.g. nlist.h). This suggests that early in 1982 the merge between 4.1 code and 32V code took place, to create the foundation for further development (“proto 8th edition”, so to speak).

Similarly, there are a dozen or so files in the kernel that all have a file date of November 1982. The most interesting one of these is “dtline.c”, a character mode Datakit driver: it uses ‘streams’. This suggests that there was a further code merge late in 1982 and implies that ‘streams’ were developed prior to that date.

From the S/F-Unix papers it seems that ‘streams’ did not exist in 1981, at least they are not mentioned in an otherwise comprehensive set of papers. On the other hand, the S/F-Unix work was done in the Exploratory group, not the Research group: maybe it was inappropriate to mention.

All in all, my hypotheses would be that:
- the 32V/4.1 merge took place early in 1982 
- ‘streams’ were developed in 1982 on 32V (maybe also V7) systems
- a further merge took place late in 1982 that combined the new base with latest developments

Does that sound correct, or was it all different?

Related is the question when the "file system switch" was added. It must have been later than 1981 and before 1985, but I have not been able to pinpoint it further.

Paul

Re: [TUHS] 8th Edition timeline

[arnold]

Paul Ruizendaal <pnr@planet.nl> wrote:

> Related is the question when the "file system switch" was added. It must
> have been later than 1981 and before 1985, but I have not been able to
> pinpoint it further.

IIRC there was a "paper" (only an abstract) on the file system
switch published in a USENIX conference proceedings. That woud help
trace it down.

Peter Weinberger, who did it, is at Google; you could ask him
directly, as well.

Arnold

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

On 29 Mar 2020, at 16:04, arnold@skeeve.com wrote:
> 
> Paul Ruizendaal <pnr@planet.nl> wrote:
> 
>> Related is the question when the "file system switch" was added. It must
>> have been later than 1981 and before 1985, but I have not been able to
>> pinpoint it further.
> 
> IIRC there was a "paper" (only an abstract) on the file system
> switch published in a USENIX conference proceedings. That woud help
> trace it down.

I have that paper (“The Unix 8th Edition Network File System”), it was presented at a March 1985 ACM conference. However, there are indications that the roots of the file system switch existed earlier, possibly much earlier.

I think Doug McIlroy once described 1973 as a pivotal year for Unix, with many concepts devised that would blossom in the following 3-5 years. I’m increasingly tempted to think that Summer ’81 - Summer ’82 was a similarly pivotal year.

> Peter Weinberger, who did it, is at Google; you could ask him
> directly, as well.

That is a good idea. If someone has the email address I’d appreciate an off list message.

Paul

Re: [TUHS] 8th Edition timeline

[Rob Pike]

I have a dog in the fight, having joined in June 1980, but that is not
a coincidence. The period of 1980-1982 was a big one for 127 (soon
1127) as they were finally given the chance to grow, and I was one of
the lucky early hires in that burst. New blood brought in new ideas
and things happened fast.

It was also the time of the VAX; the center's 11/780 arrived in late
1980 I think, maybe early 1981. Our first experiments with graphical
terminals spanned 1980 to early 1981, using Greg Chesson's mux, but by
late 1981 we were using Dennis's streams (only STREAMS when they went
to USG) and the select system call, which was by then running in a
merged Berkeley/Research Unix that eventually became the Eighth
Edition.

My notebooks can probably lock down a lot of this as I was a prolific
note-taker back then, when they still made paper.

-rob



On Mon, Mar 30, 2020 at 5:14 AM Paul Ruizendaal <pnr@planet.nl> wrote:
>
> On 29 Mar 2020, at 16:04, arnold@skeeve.com wrote:
> >
> > Paul Ruizendaal <pnr@planet.nl> wrote:
> >
> >> Related is the question when the "file system switch" was added. It must
> >> have been later than 1981 and before 1985, but I have not been able to
> >> pinpoint it further.
> >
> > IIRC there was a "paper" (only an abstract) on the file system
> > switch published in a USENIX conference proceedings. That woud help
> > trace it down.
>
> I have that paper (“The Unix 8th Edition Network File System”), it was presented at a March 1985 ACM conference. However, there are indications that the roots of the file system switch existed earlier, possibly much earlier.
>
> I think Doug McIlroy once described 1973 as a pivotal year for Unix, with many concepts devised that would blossom in the following 3-5 years. I’m increasingly tempted to think that Summer ’81 - Summer ’82 was a similarly pivotal year.
>
> > Peter Weinberger, who did it, is at Google; you could ask him
> > directly, as well.
>
> That is a good idea. If someone has the email address I’d appreciate an off list message.
>
> Paul
>
>
>

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

> On 29 Mar 2020, at 23:48, Rob Pike <robpike@gmail.com> wrote:
> 
> I have a dog in the fight, having joined in June 1980, but that is not
> a coincidence. The period of 1980-1982 was a big one for 127 (soon
> 1127) as they were finally given the chance to grow, and I was one of
> the lucky early hires in that burst. New blood brought in new ideas
> and things happened fast.

I had not realised that the Research group expanded in 1980, but it fits.

> It was also the time of the VAX; the center's 11/780 arrived in late
> 1980 I think, maybe early 1981.

I did realise that bit, and it made me wonder if the ’73 burst was in part driven by the arrival of a 11/45.

> Our first experiments with graphical terminals spanned 1980 to early 1981,

Yes, as you may remember from this list I dove into that last November - schematics, tools & firmware.

> using Greg Chesson's mux, 

Chesson’s MPX files remain a puzzle piece that is somewhat difficult to fit in the overall story, having so many aspects. It sits between Rand ports and SysIII fifo’s, experiments with non-blocking I/O, has aspects of pseudo-terminals, etc. I have not been able to figure out what immediate need they served, unless it was used in the first generation Datakit software (as MPX precedes the Jerq, that cannot have been the immediate need.)

> but by late 1981 we were using Dennis's streams (only STREAMS when
> they went to USG) and the select system call, which was by then running
> in a merged Berkeley/Research Unix that eventually became the Eighth
> Edition.

To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code was ready in June 1981 and the ’select’ system call was first proposed in July 1981, so it is possible. However, in the BSD line ’select’ was not fully implemented until March/April 1982.

It is certainly possible, even likely, that ‘streams’ date from 1981 or earlier. Networks don’t mesh well with TTY line disciplines and clist buffering - that pain will have become apparent already in 1979. Maybe it was among the first things to be fixed when the VAX arrived.

> My notebooks can probably lock down a lot of this as I was a prolific
> note-taker back then, when they still made paper.

If someday you have time for this, it would be much appreciated!

Paul

Re: [TUHS] 8th Edition timeline

[Rob Pike]

I've looked through my notes and unfortunately there's very little
about this as the notes are mostly about graphics and physics.

You're right I was a little early above, but the "merge" was not an
event. It was a process that went on for months, years even.

Dennis wrote the select we used, I'm almost certain of that. We (he
and I) talked about its design at the time, after a trip he made to
Berkeley about the DARPA Stuff. He had had many discussions with the
BSD work going on and wanted to couple it to the streams work to make
my graphical multiplexer work. Mpx was killing me (and killing the
kernel), but I'm nearly certain that I was using select (with our form
of pseudo-ttys) before or by very early 1982. I'm all but certain; we
gave up on mpx(2) very early.

Streams came later; Dennis's Show and Tell about them was early in 1982.

I'm not sure, but we may have booted London & Reiser's VAX port first,
but we moved to Berkeley code pretty early. We (Dennis mostly, but
others) were talking to Berkeley often. We didn't just wait for a
release; we were a part (not sure I could say now how important, but a
part) of it from early on.


-rob

On Mon, Mar 30, 2020 at 6:44 PM Paul Ruizendaal <pnr@planet.nl> wrote:
>
>
> > On 29 Mar 2020, at 23:48, Rob Pike <robpike@gmail.com> wrote:
> >
> > I have a dog in the fight, having joined in June 1980, but that is not
> > a coincidence. The period of 1980-1982 was a big one for 127 (soon
> > 1127) as they were finally given the chance to grow, and I was one of
> > the lucky early hires in that burst. New blood brought in new ideas
> > and things happened fast.
>
> I had not realised that the Research group expanded in 1980, but it fits.
>
> > It was also the time of the VAX; the center's 11/780 arrived in late
> > 1980 I think, maybe early 1981.
>
> I did realise that bit, and it made me wonder if the ’73 burst was in part driven by the arrival of a 11/45.
>
> > Our first experiments with graphical terminals spanned 1980 to early 1981,
>
> Yes, as you may remember from this list I dove into that last November - schematics, tools & firmware.
>
> > using Greg Chesson's mux,
>
> Chesson’s MPX files remain a puzzle piece that is somewhat difficult to fit in the overall story, having so many aspects. It sits between Rand ports and SysIII fifo’s, experiments with non-blocking I/O, has aspects of pseudo-terminals, etc. I have not been able to figure out what immediate need they served, unless it was used in the first generation Datakit software (as MPX precedes the Jerq, that cannot have been the immediate need.)
>
> > but by late 1981 we were using Dennis's streams (only STREAMS when
> > they went to USG) and the select system call, which was by then running
> > in a merged Berkeley/Research Unix that eventually became the Eighth
> > Edition.
>
> To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code was ready in June 1981 and the ’select’ system call was first proposed in July 1981, so it is possible. However, in the BSD line ’select’ was not fully implemented until March/April 1982.
>
> It is certainly possible, even likely, that ‘streams’ date from 1981 or earlier. Networks don’t mesh well with TTY line disciplines and clist buffering - that pain will have become apparent already in 1979. Maybe it was among the first things to be fixed when the VAX arrived.
>
> > My notebooks can probably lock down a lot of this as I was a prolific
> > note-taker back then, when they still made paper.
>
> If someday you have time for this, it would be much appreciated!
>
> Paul
>

Re: [TUHS] 8th Edition timeline

[emanuel stiebler]

On 2020-03-30 05:06, Rob Pike wrote:
> I've looked through my notes and unfortunately there's very little
> about this as the notes are mostly about graphics and physics.

I wouldn't mind seeing the note baout graphics, even if not on topic for
this group ...

Re: [TUHS] 8th Edition timeline

[Noel Hunt]

[-- Attachment #1: Type: text/plain, Size: 618 bytes --]

I would be interested too. That was a seminal time with the
proliferation of graphical programs, such as jim, pads/pi,
proof, the sophisticated menus of 'mhit.c' etc. It is curious
that little if any of that code made the transition to Plan9
(apart from sam, nee jim).


On Sun, Apr 5, 2020 at 12:19 AM emanuel stiebler <emu@e-bbes.com> wrote:

> On 2020-03-30 05:06, Rob Pike wrote:
> > I've looked through my notes and unfortunately there's very little
> > about this as the notes are mostly about graphics and physics.
>
> I wouldn't mind seeing the note baout graphics, even if not on topic for
> this group ...
>

[-- Attachment #2: Type: text/html, Size: 1372 bytes --]

Re: [TUHS] 8th Edition timeline

[Rob Pike]

Sam wasn't nee jim. Jim was a toy, sam is a serious editor.

My notebooks are hundreds of pages of figuring stuff out. Perhaps
valuable information to historians, but not easily compressed for this
forum.

-rob

On Sun, Apr 5, 2020 at 5:57 AM Noel Hunt <noel.hunt@gmail.com> wrote:
>
> I would be interested too. That was a seminal time with the
> proliferation of graphical programs, such as jim, pads/pi,
> proof, the sophisticated menus of 'mhit.c' etc. It is curious
> that little if any of that code made the transition to Plan9
> (apart from sam, nee jim).
>
>
> On Sun, Apr 5, 2020 at 12:19 AM emanuel stiebler <emu@e-bbes.com> wrote:
>>
>> On 2020-03-30 05:06, Rob Pike wrote:
>> > I've looked through my notes and unfortunately there's very little
>> > about this as the notes are mostly about graphics and physics.
>>
>> I wouldn't mind seeing the note baout graphics, even if not on topic for
>> this group ...

Re: [TUHS] 8th Edition timeline

[Noel Hunt]

[-- Attachment #1: Type: text/plain, Size: 292 bytes --]

> Sam wasn't nee jim. Jim was a toy, sam is a serious editor.

I don't doubt that, as a long-time user of sam (and jim
initially, 30 or so years ago), but it is interesting to
see the gradual sophistication of, say, the frame library
presumably in response to changes in hardware, and so on.

[-- Attachment #2: Type: text/html, Size: 823 bytes --]

Re: [TUHS] 8th Edition timeline

[aksr]

On Sun, Apr 05, 2020 at 07:32:22AM +1000, Rob Pike wrote:
> Sam wasn't nee jim. Jim was a toy, sam is a serious editor.
>
> My notebooks are hundreds of pages of figuring stuff out. Perhaps
> valuable information to historians, but not easily compressed for this
> forum.

I think this info would be more than interesting.

Maybe if and when you have the time,
you could compress this (at least for your blog)?

Re: [TUHS] 8th Edition timeline

[emanuel stiebler]

On 2020-04-04 17:32, Rob Pike wrote:
> Sam wasn't nee jim. Jim was a toy, sam is a serious editor.
> 
> My notebooks are hundreds of pages of figuring stuff out. Perhaps
> valuable information to historians, but not easily compressed for this
> forum.

If you need anybody with a scanner an patience, please tell me ;-)

> -rob
> 
> On Sun, Apr 5, 2020 at 5:57 AM Noel Hunt <noel.hunt@gmail.com> wrote:
>>
>> I would be interested too. That was a seminal time with the
>> proliferation of graphical programs, such as jim, pads/pi,
>> proof, the sophisticated menus of 'mhit.c' etc. It is curious
>> that little if any of that code made the transition to Plan9
>> (apart from sam, nee jim).
>>
>>
>> On Sun, Apr 5, 2020 at 12:19 AM emanuel stiebler <emu@e-bbes.com> wrote:
>>>
>>> On 2020-03-30 05:06, Rob Pike wrote:
>>>> I've looked through my notes and unfortunately there's very little
>>>> about this as the notes are mostly about graphics and physics.
>>>
>>> I wouldn't mind seeing the note baout graphics, even if not on topic for
>>> this group ...

Re: [TUHS] 8th Edition timeline

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2691 bytes --]

Paul - see below..

On Mon, Mar 30, 2020 at 3:43 AM Paul Ruizendaal <pnr@planet.nl> wrote:

> To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code
> was ready in June 1981 and the ’select’ system call was first proposed in
> July 1981, so it is possible. However, in the BSD line ’select’ was not
> fully implemented until March/April 1982.
>
Please be careful here. 4.1BSD is different from the pre-'4.2' released and
4.2BSD itself. 4.1aBSD was the first pseudo release[1] that started to have
the major surgery to support Bill's sockets idea and splice in the UCB
rewrite of the BBN code.  4.1BSD was the first system for the Vax that
really wide distribution.  'Anyone' with an AT&T license could get it and
most people did.  Remember this is the system that BBN (Gurwitz) did the
original IP/TCP support (sans sockets - i.e. /dev/ip /dev/tcp ...).  Your
date of June '81 for the 4.1BSD release seems late, but I'll accept it.
3BSD was 1979, and I thought 4BSD was a year later, with 4.1BSD a few
months after 4BSD (few people actually got 4BSD)

That said, Bill and Sam did the heavy lifting on select(2) first in 4.1aBSD
and there were some issues (again I have forgotten the details -- I do
remember, I was working on my thesis and I had a do a huge rewrite of the
AP kernel support to handle select(2) properly).  I remember talking to Sam
(arguing with him most probably) about it one night before it was fully
created.   I want to say, he had worked on something similar at the firm he
was at (the firm name I now forget -- si-mumble -- they were in Mt. View)
before he joined CRSG.  I don't remember now the issues I had, but I do
remember it was a bit of mess to support the way the AP hardware assumed it
could do DMA on the UBA[2]


[1] 4.1a/4.1b/4.1cBSD was officially internal to UCB and some ARPA-sites,
although I would have expected someone like Dennis in 1127 to have been
sent it also, as wnj was in the process of leaving for Sun and he took them
with him.  For instance, I would take 4.1c to Masscomp.  The key is that
these were not as widely distributed as 4.1BSD.   4.2BSD would really
accelerate BSD UNIX uptake, because of the networking support but there was
more than 2-3 years between 4.1BSD and 4.2BSD.

 [2] The AP's MMU/DMA interface at the time, was causing me great hair, and
that was likely to have been part of the reason I wanted some help/changes
in the KPI interface - which is actually funny, they eventually came with
the CMU Mach MMU changes of 4.4BSD (which was much more friendly to a
multiprocessor/coprocessor architecture).  FWIW: I never got them when I
was at UCB.

[-- Attachment #2: Type: text/html, Size: 4025 bytes --]

Re: [TUHS] 8th Edition timeline

[reed]

On Mon, 30 Mar 2020, Clem Cole wrote:

...
> kernel support to?handle?select(2) properly).? I remember talking to Sam (arguing
> with him most probably) about it one night before it was fully created.? ?Iwant
> to say, he had worked on something similar at the firm he was at (the firm
> name I now forget -- si-mumble -- they were in Mt. View) before he joined
> CRSG.? I don't remember now the issues I had, but I do remember it was a bit
> of mess to support the way the AP hardware assumed it could do DMA on the
> UBA[2]

From my notes:

After graduating, Leffler got a job in the Silicon Valley
at Sytek, a small firm in Sunnyvale.
% CITE: archives/csrg-archives/disk1/mnt/2.79/usenet/netdir/sites/sytek
They used a PDP 11/45 and moved to a VAX 11/750 running VM/UNIX
with four 300 baud ACU's (Automatic Call Units aka ``modems``) ---
they marketed a broadband network to
connect thousands of stations using coaxial cable over a distance
of 40 miles.
% maybe CITE: 
http://books.google.com/books?id=AyZfLaxYr9YC&pg=PA43&lpg=PA43&dq=
%28408%29+734-9000++sytek&source=bl&ots=SYGI9rBAOV&sig=rxBo1SgfqdTLuNotXaa59nrR6
C4&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CEAQ6AEwBA#v=onepage&q=%28408%29%20
734-9000%20%20sytek&f=false
Since they were using a VAX and he had kept in touch with Joy, he was
a beta-test site for 4BSD.\cite{salus1994}

% NOTE: about sytek: an ad to find employees
%http://books.google.com/books?id=ChMAmfS1nEkC&pg=PA85&lpg=PA85&dq=%28408%29+734
-9000++sytek&source=bl&ots=aS_yQznXAy&sig=6vA7NTbspu3gcf_jT2sOiUSRPZU&hl=en&sa=X
&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CC4Q6AEwAA#v=onepage&q=%28408%29%20734-9000%20%2
0sytek&f=false

% NOTE: not to take over since no Sun then
Leffler went to Berkeley to work with Joy\cite{billshannon2} in the
fall of 1981.\cite{salus1994}.

Re: [TUHS] 8th Edition timeline

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2639 bytes --]

Jeremy -- Sytek - that was it -- thank you.   Sunnyvale/Mt. View -- who can
tell the difference?  ;-)

Thanks again,
Clem

On Mon, Mar 30, 2020 at 10:21 AM <reed@reedmedia.net> wrote:

> On Mon, 30 Mar 2020, Clem Cole wrote:
>
> ...
> > kernel support to?handle?select(2) properly).? I remember talking to Sam
> (arguing
> > with him most probably) about it one night before it was fully created.?
> ?Iwant
> > to say, he had worked on something similar at the firm he was at (the
> firm
> > name I now forget -- si-mumble -- they were in Mt. View) before he joined
> > CRSG.? I don't remember now the issues I had, but I do remember it was a
> bit
> > of mess to support the way the AP hardware assumed it could do DMA on the
> > UBA[2]
>
> From my notes:
>
> After graduating, Leffler got a job in the Silicon Valley
> at Sytek, a small firm in Sunnyvale.
> % CITE: archives/csrg-archives/disk1/mnt/2.79/usenet/netdir/sites/sytek
> They used a PDP 11/45 and moved to a VAX 11/750 running VM/UNIX
> with four 300 baud ACU's (Automatic Call Units aka ``modems``) ---
> they marketed a broadband network to
> connect thousands of stations using coaxial cable over a distance
> of 40 miles.
> % maybe CITE:
> http://books.google.com/books?id=AyZfLaxYr9YC&pg=PA43&lpg=PA43&dq=
>
> %28408%29+734-9000++sytek&source=bl&ots=SYGI9rBAOV&sig=rxBo1SgfqdTLuNotXaa59nrR6
>
> C4&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CEAQ6AEwBA#v=onepage&q=%28408%29%20
> 734-9000%20%20sytek&f=false
> <http://books.google.com/books?id=AyZfLaxYr9YC&pg=PA43&lpg=PA43&dq=%28408%29+734-9000++sytek&source=bl&ots=SYGI9rBAOV&sig=rxBo1SgfqdTLuNotXaa59nrR6C4&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CEAQ6AEwBA#v=onepage&q=%28408%29%20734-9000%20%20sytek&f=false>
> Since they were using a VAX and he had kept in touch with Joy, he was
> a beta-test site for 4BSD.\cite{salus1994}
>
> % NOTE: about sytek: an ad to find employees
> %
> http://books.google.com/books?id=ChMAmfS1nEkC&pg=PA85&lpg=PA85&dq=%28408%29+734
>
> -9000++sytek&source=bl&ots=aS_yQznXAy&sig=6vA7NTbspu3gcf_jT2sOiUSRPZU&hl=en&sa=X
>
> &ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CC4Q6AEwAA#v=onepage&q=%28408%29%20734-9000%20%2
> 0sytek&f=false
> <http://books.google.com/books?id=ChMAmfS1nEkC&pg=PA85&lpg=PA85&dq=%28408%29+734-9000++sytek&source=bl&ots=aS_yQznXAy&sig=6vA7NTbspu3gcf_jT2sOiUSRPZU&hl=en&sa=X&ei=CzbwUJTjI6HN2QX48oGIAw&ved=0CC4Q6AEwAA#v=onepage&q=%28408%29%20734-9000%20%20sytek&f=false>
>
> % NOTE: not to take over since no Sun then
> Leffler went to Berkeley to work with Joy\cite{billshannon2} in the
> fall of 1981.\cite{salus1994}.
>

[-- Attachment #2: Type: text/html, Size: 3589 bytes --]

[TUHS] Sytek (was Re: 8th Edition timeline)

[Jon Forrest]

On 3/30/2020 7:41 AM, Clem Cole wrote:
> Jeremy -- Sytek - that was it -- thank you.   Sunnyvale/Mt. View -- who 
> can tell the difference?  ;-)

Here's a story about Sytek (I hope I haven't told it before - I'm
starting to forget who I've told stories to).

In 1977/1978 I was an extremely junior engineeer at Ford Aerospace.
I was in the group that was both acting as the main Unix promoter
in Ford, and also was making a provably-secure version of Unix
called KSOS. The group had a bunch of excellent senior people (e.g.
John Nagle, Tom Berson, Ken Biba, Mike Pliner, Mike Padlipsky).

This was my first real job in the software industry so I didn't know
what to expect. After about 6 months I noticed that there were a lot
of closed-door meetings going on. I thought this was normal. One day
I was walking by an office when the door opened up and I was invited
in. I didn't know if I had done something wrong and was about to be
fired, or what.

I turned out that I had nothing to worry about. What was happening
was that some of the people in the group were going to leave Ford
to start their own company. They needed some junior people also,
and they asked me to join them. I gave it some thought but turned
them down because I didn't want to live in Silicon Valley anymore,
and wanted to return to Santa Barabara where I belonged.

The company that they started was Sytek, which got the contract
from IBM to create NetBIOS. They also did other networking work
that has been mentioned in this thread.

If I had gone to Sytex I would have been employe #8 with lots of
stock options. I often wonder how my life would have been different
if I had done this. I suspect many people on TUHS have similar stories
about the ones that got away. In fact, this might be a good topic
for a new thread.

Cordially,
Jon Forrest

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

Comments in line:

> On 30 Mar 2020, at 15:26, Clem Cole <clemc@ccc.com> wrote:
> 
> Paul - see below..
> 
> On Mon, Mar 30, 2020 at 3:43 AM Paul Ruizendaal <pnr@planet.nl> wrote:
> To be honest, late 1981 sounds a bit too early for the merge. The 4.1 code was ready in June 1981 and the ’select’ system call was first proposed in July 1981, so it is possible. However, in the BSD line ’select’ was not fully implemented until March/April 1982.

> Please be careful here. 4.1BSD is different from the pre-'4.2' released and 4.2BSD itself. 4.1aBSD was the first pseudo release[1] that started to have the major surgery to support Bill's sockets idea and splice in the UCB rewrite of the BBN code.  4.1BSD was the first system for the Vax that really wide distribution.  'Anyone' with an AT&T license could get it and most people did.  Remember this is the system that BBN (Gurwitz) did the original IP/TCP support (sans sockets - i.e. /dev/ip /dev/tcp ...).  Your date of June '81 for the 4.1BSD release seems late, but I'll accept it.  3BSD was 1979, and I thought 4BSD was a year later, with 4.1BSD a few months after 4BSD (few people actually got 4BSD)

I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD and June 1981 for 4.1BSD. From the SCCS log I’d say November 9th and July 1st respectively.

As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in April 1981 - most of that is in a separate tree, with a few #ifdef’s in the main tree. From SCCS it seems that implementation work on sockets started in Oct/Nov 1981 and 4.1a was released in April or March 1982. From the partial coverage in SCCS I’d say there is no BBN code in 4.1a anymore. The socket API in that release was different from what ended being in 4.1c and 4.2, and seems to have had some wider visibility in 2.9BSD and UniPlus System V - both of which used the 4.1a API (and probably code base).

> That said, Bill and Sam did the heavy lifting on select(2) first in 4.1aBSD and there were some issues (again I have forgotten the details -- I do remember, I was working on my thesis and I had a do a huge rewrite of the AP kernel support to handle select(2) properly).  I remember talking to Sam (arguing with him most probably) about it one night before it was fully created.   I want to say, he had worked on something similar at the firm he was at (the firm name I now forget -- si-mumble -- they were in Mt. View) before he joined CRSG.  I don't remember now the issues I had, but I do remember it was a bit of mess to support the way the AP hardware assumed it could do DMA on the UBA[2]

> [1] 4.1a/4.1b/4.1cBSD was officially internal to UCB and some ARPA-sites, although I would have expected someone like Dennis in 1127 to have been sent it also, as wnj was in the process of leaving for Sun and he took them with him.  For instance, I would take 4.1c to Masscomp.  The key is that these were not as widely distributed as 4.1BSD.   4.2BSD would really accelerate BSD UNIX uptake, because of the networking support but there was more than 2-3 years between 4.1BSD and 4.2BSD.

Yes, Summer 81 - Summer 82 must have been a vibrant year, with much interaction between various Unix groups. By the way, dmr was on the 4.2BSD steering group.

>  [2] The AP's MMU/DMA interface at the time, was causing me great hair, and that was likely to have been part of the reason I wanted some help/changes in the KPI interface - which is actually funny, they eventually came with the CMU Mach MMU changes of 4.4BSD (which was much more friendly to a multiprocessor/coprocessor architecture).  FWIW: I never got them when I was at UCB.

Re: [TUHS] 8th Edition timeline

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 4095 bytes --]

On Mon, Mar 30, 2020 at 12:02 PM Paul Ruizendaal <pnr@planet.nl> wrote:

> Comments in line:
>
> > On 30 Mar 2020, at 15:26, Clem Cole <clemc@ccc.com> wrote:
> Your date of June '81 for the 4.1BSD release seems late, but I'll accept
> it.  3BSD was 1979, and I thought 4BSD was a year later, with 4.1BSD a few
> months after 4BSD (few people actually got 4BSD)
>
> I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD
> and June 1981 for 4.1BSD. From the SCCS log I’d say November 9th and July
> 1st respectively.

Ok, that sounds right 6-7 months between them.   That was the time of the
'FASTVAX' work and the fight with Stanford over if BSD or VMS was to be the
official Arpa OS.  I remember Joy was incredibly prolific during that
time.  From a user standpoint, 4BSD and 4.1BSD are really similar, but he
was hacking the kernel.  It seemed like a new change came out all the
time.  The CAD group (where I was) always wanted the fastest system, so
there was pressure on me to follow him, but I was trying to make progress
on the AP work (my thesis) so I was a little loath to take the churn.



>
>
> As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in
> April 1981 - most of that is in a separate tree, with a few #ifdef’s in the
> main tree.

I'm pretty sure that was mostly Eric Cooper (ecc) IIRC.  We had a tape
directly from Gurwitz running in the CAD lab in Cory Hall, running on Xerox
3M Ethernet and because we needed it between our three 780s systems. We had
ordered the 3C501's from 3COM and we were going to be the Beta for the
Interlan board - which I wrote the original driver and gave it to Sam.  I'm
not sure when IngVAX arrived, Eric Schmidt had Ing70 downstairs (and that
was the ArpaNet connect via a VDH up the hill to the IMP at LBL.  We had
been running BerkNET over serial lines.  Ernie and Kim were the two 780s
over in Evan's (the big pile of 750s and the C30 IMP were still in the
future).




> From SCCS it seems that implementation work on sockets started in Oct/Nov
> 1981 and 4.1a was released in April or March 1982.

That sounds about right.  It was a bunch of wnj marathon hacking sessions.



> From the partial coverage in SCCS I’d say there is no BBN code in 4.1a
> anymore.

Mumble ...   I sort of have a hard time with that.  I knew and worked with
both of them in my day and have heard both sides of the story. Joy had
Gurwitz's code and was hacking it - that is just a fact.  Claiming pure
authorship is a tad extreme.   For instance, Andy VanDam told me that the
whole mbuf's stuff is something Rob originally wrote for a project for him
at Brown when he was an undergrad.  Rob brought it BBN and he used it for
the OS-independent TCP (i.e. the HP3000 version) originally and it landed
in the UNIX version.  Joy hacked on it, but that's a data structure
directly from Rob.    On the other hand, Joy types open curly brace, close
curly brace, and hacks in code the fastest of anyone I ever knew (someone,
maybe Mary Ann Horton, once said he 'wrote code at 9600').  But if you look
at it, its sometimes not pretty.  It works and its usually fast, which is
hard to argue with.




> The socket API in that release was different from what ended being in 4.1c
> and 4.2,

Definitely.   I thought other hands besides Joy started to mess with it
(like Sam)



> and seems to have had some wider visibility in 2.9BSD and UniPlus System V
> - both of which used the 4.1a API (and probably code base).

Yeah, Keith was taking things was Sam at pretty hectic pace trying to keep
the 11's running.    I would guess that took more of a 'good enough'
attitude, they got the TCP stack in the limited address space and did not
want to try much more

>
>
> Yes, Summer 81 - Summer 82 must have been a vibrant year, with much
> interaction between various Unix groups.

It was a fun time to be apart of it all.



> By the way, dmr was on the 4.2BSD steering group.

Yes, I know.   That's how I first got to know him.

[-- Attachment #2: Type: text/html, Size: 6846 bytes --]

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

> [...] the whole mbuf's stuff is something [...]

I would prefer not to go into the BSD code history at this point in time, as I’d like to stay focused on what was going on at Bell Labs between 7th and 8th edition, as far as networking is concerned. Of course it was all interconnecting, and the topic of mbuf's is a nice arc to take us ultimately back too streams.

Looking at network buffering in Unix, I think the story begins in 1973 with Spider. In Spider, the network driver was a character mode device, but used disk block buffering for storage. This worked well, because the Spider interface sent/received data through DMA, in cells of 256 bytes and at speeds comparable to contemporary disks:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=V4/nsys/dmr/tdir/tiu.c.

In 1975 the UoI guys doing Arpa Unix had to interface to an IMP, which was much slower than a disk but much faster than a terminal. They concluded that neither clists not disk buffers were a good match and Steve Bunch came up with ’netbuf’s, which were 64 byte blocks (carved out of a pool of disk blocks). Later (1978?) this code was modified to optionally use separate buffer heads and buffer data, as part of the UoC buffer mod’s (to alleviate the pressure on a limited address space):
https://minnie.tuhs.org/cgi-bin/utree.pl?file=SRI-NOSC/h/net/netbuf.h

In 1979 the first Datakit software was done (Chesson), where I don’t have much information. My hypothesis is that this used clists, but I’m still trying to find a copy of his paper about this implementation. Maybe the driver used a precursor to a streams like data structure. Maybe this is also where MPX files fit in. Unfortunately, currently I just don’t know.

In 1980 new TCP software for the VAX is being written, and again it needs to find a data buffer approach. This is ‘mbuf’s, which are similar to the earlier ’netbuf’s. The main changes are that the size is increased to 128 bytes and that a start offset is added, which makes it possible to remove headers without copying the buffer. The ‘memberof’ field is replaced by a second linkage pointer. The UoC idea to separate the header and data is dropped again. They (mbufs) may have other origins as well, but as far as Unix networking is concerned they are an evolution of the Arpa Unix ‘netbuf’s:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=BBN-Vax-TCP/bbnnet/mbuf.h

Then we get to (late) 1981. On the Labs side streams are created, in my current understanding mainly to support networking and in particular Datakit. The difference with Spider is that the Datakit interface was not DMA based and that the data arrives in 9 bit envelopes instead of 8 bit bytes. The 9th bit distinguishes between data bytes and control bytes. Again neither clists nor disk buffers are a good match for buffering and a new solution is created. Looking at buffers, these consist of a header and a separate datablock. The datablock can be of various sizes: 4, 16, 64 and 1024 bytes - a match with the varying nature of network traffic. Each block has a type (a.o. DATA and CONTROL) which has various uses, but certainly gives a home to the 9th Datakit bit where needed:
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/include/sys/stream.h
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/sys/dev/stream.c

Over on the west coast the mbuf structure is extended with a new field, “type” at the same time, and it transitions from being a pure network data buffer to a more generic buffer. It remains at a fixed size of 128 bytes with inline data; the option to store data in a separate block of varying size will only come back late in 4.3BSD. For the 4.2 version:
https://minnie.tuhs.org/cgi-bin/utree.pl?file=4.2BSD/usr/src/sys/h/mbuf.h

When viewed at this code level it would seem to me that by various routes everybody had arrived at more or less the same the point in network buffering approach by 1981. That has some interesting implications. At this level, is there really a difference between a pipe implemented as cross connected streams and a pipe implemented as cross connected sockets? Is an unbound socket not about the same as an unconnected stream head?

But before getting into such discussions, I’d really like to get the development story from 7th to 8th edition as clear and precise as possible after 40 years.




> On 30 Mar 2020, at 21:31, Clem Cole <clemc@ccc.com> wrote:
> 
> 
> 
> On Mon, Mar 30, 2020 at 12:02 PM Paul Ruizendaal <pnr@planet.nl> wrote:
> Comments in line:
> 
> > On 30 Mar 2020, at 15:26, Clem Cole <clemc@ccc.com> wrote:
> Your date of June '81 for the 4.1BSD release seems late, but I'll accept it.  3BSD was 1979, and I thought 4BSD was a year later, with 4.1BSD a few months after 4BSD (few people actually got 4BSD)
> 
> I am aware of that. “20 years of Berkely Unix” says November 1980 for 4BSD and June 1981 for 4.1BSD. From the SCCS log I’d say November 9th and July 1st respectively.
> Ok, that sounds right 6-7 months between them.   That was the time of the 'FASTVAX' work and the fight with Stanford over if BSD or VMS was to be the official Arpa OS.  I remember Joy was incredibly prolific during that time.  From a user standpoint, 4BSD and 4.1BSD are really similar, but he was hacking the kernel.  It seemed like a new change came out all the time.  The CAD group (where I was) always wanted the fastest system, so there was pressure on me to follow him, but I was trying to make progress on the AP work (my thesis) so I was a little loath to take the churn.
> 
>  
> 
> 
> As far as I can tell CSRG integrated the BBN stack with (almost) 4.1 in April 1981 - most of that is in a separate tree, with a few #ifdef’s in the main tree.
> I'm pretty sure that was mostly Eric Cooper (ecc) IIRC.  We had a tape directly from Gurwitz running in the CAD lab in Cory Hall, running on Xerox 3M Ethernet and because we needed it between our three 780s systems. We had ordered the 3C501's from 3COM and we were going to be the Beta for the Interlan board - which I wrote the original driver and gave it to Sam.  I'm not sure when IngVAX arrived, Eric Schmidt had Ing70 downstairs (and that was the ArpaNet connect via a VDH up the hill to the IMP at LBL.  We had been running BerkNET over serial lines.  Ernie and Kim were the two 780s over in Evan's (the big pile of 750s and the C30 IMP were still in the future).
> 
> 
>  
> From SCCS it seems that implementation work on sockets started in Oct/Nov 1981 and 4.1a was released in April or March 1982.
> That sounds about right.  It was a bunch of wnj marathon hacking sessions.
> 
>  
> From the partial coverage in SCCS I’d say there is no BBN code in 4.1a anymore.
> Mumble ...   I sort of have a hard time with that.  I knew and worked with both of them in my day and have heard both sides of the story. Joy had Gurwitz's code and was hacking it - that is just a fact.  Claiming pure authorship is a tad extreme.   For instance, Andy VanDam told me that the whole mbuf's stuff is something Rob originally wrote for a project for him at Brown when he was an undergrad.  Rob brought it BBN and he used it for the OS-independent TCP (i.e. the HP3000 version) originally and it landed in the UNIX version.  Joy hacked on it, but that's a data structure directly from Rob.    On the other hand, Joy types open curly brace, close curly brace, and hacks in code the fastest of anyone I ever knew (someone, maybe Mary Ann Horton, once said he 'wrote code at 9600').  But if you look at it, its sometimes not pretty.  It works and its usually fast, which is hard to argue with.
> 
> 
>  
> The socket API in that release was different from what ended being in 4.1c and 4.2,
> Definitely.   I thought other hands besides Joy started to mess with it (like Sam)
> 
>  
> and seems to have had some wider visibility in 2.9BSD and UniPlus System V - both of which used the 4.1a API (and probably code base).
> Yeah, Keith was taking things was Sam at pretty hectic pace trying to keep the 11's running.    I would guess that took more of a 'good enough' attitude, they got the TCP stack in the limited address space and did not want to try much more
> 
> 
> Yes, Summer 81 - Summer 82 must have been a vibrant year, with much interaction between various Unix groups.
> It was a fun time to be apart of it all.
> 
>  
> By the way, dmr was on the 4.2BSD steering group.
> Yes, I know.   That's how I first got to know him. 

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

As was suggested on the list, I’ve reached out to Peter Weinberger to better understand the time line of the File System Switch and the 8th edition network file system. He has been very helpful, but the one line summary is that it is unfortunately too long ago to remember specific details with any certainty. In general Peter remembers that he was concerned that the project was too big for one person to do, and hence always looked for design choices that would leave the work scope manageable.

Time line.

Since my last post on this subject I have found that the ACM conference talk of March 1985 was also held 9 months earlier at a Usenix conference - leaving a time slot between end of 1981 and summer 1984. Peter vaguely remembers that the essential ideas were done "before 1983”. It would stand to reason that 1983 was spent on getting corner cases of the network file system right, but all this is no more than plausible conjecture.

File system switch (FSS).

The guiding thought for the FSS was to extend the philosophy of ‘everything is a file’ to new areas, also other than network files. Early implementations already included a simple, read-only ‘/proc’ file system for example. I asked if any experiments had been done with virtualising ‘/dev', but Peter could not recall any such work.

I personally find that the FSS has an elegance that fits with other parts of Research Unix and asked Peter about its origins. He does not recall any special "a-ha” moments, but does recall that the way it was done just felt natural to him. Other options would have included to do the switch at the sys call level (which felt too complicated) or at the block device level (which felt too limited).

I also asked about how his reworking of ’namei’ and centralising all namespace operations in that function came about (in my view it is key to a concise switch). Here, too, it is too long ago to remember any specifics, but Peter comments that he never liked to write much code and that spending time on finding ways to make the amount of coding as small and straightforward as possible would have been in character.

Eighth edition network file system

Once the FSS exists, a simplistic network file system is not hard - just do RPC to a remote server. Peter chose to do a user level file server in order to keep the work scope and complexity down to manageable levels. As highlighted in the ACM paper, the devil is in the detail of replicating all the semantics of normal local disk files. Cases like a file being kept alive if a process still has it open, the complexities of cross-mounted network files (let alone recursively mounted), handling failed connections, etc. were hard to sort out and get right.


> On 29 Mar 2020, at 20:12, Paul Ruizendaal <pnr@planet.nl> wrote:
> 
> On 29 Mar 2020, at 16:04, arnold@skeeve.com wrote:
>> 
>> Paul Ruizendaal <pnr@planet.nl> wrote:
>> 
>>> Related is the question when the "file system switch" was added. It must
>>> have been later than 1981 and before 1985, but I have not been able to
>>> pinpoint it further.
>> 
>> IIRC there was a "paper" (only an abstract) on the file system
>> switch published in a USENIX conference proceedings. That woud help
>> trace it down.
> 
> I have that paper (“The Unix 8th Edition Network File System”), it was presented at a March 1985 ACM conference. However, there are indications that the roots of the file system switch existed earlier, possibly much earlier.
> 
> I think Doug McIlroy once described 1973 as a pivotal year for Unix, with many concepts devised that would blossom in the following 3-5 years. I’m increasingly tempted to think that Summer ’81 - Summer ’82 was a similarly pivotal year.
> 
>> Peter Weinberger, who did it, is at Google; you could ask him
>> directly, as well.
> 
> That is a good idea. If someone has the email address I’d appreciate an off list message.
> 
> Paul

Re: [TUHS] 8th Edition timeline

[Thomas Paulsen]

>As was suggested on the list, I’ve reached out to Peter Weinberger to better
>understand the time line of the File System Switch and the 8th edition network
f>ile system. 

'Introduced with System V Release 3.0, the File System Switch (FSS) architecture introduced a framework under which multiple different filesystem types could coexist in parallel.'
https://www.oreilly.com/library/view/unix-filesystems-evolution/9780471456759/chap07-sec003.html

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

On Apr 8, 2020, at 8:58 PM, Thomas Paulsen <thomas.paulsen@firemail.de> wrote:
> 
>> As was suggested on the list, I’ve reached out to Peter Weinberger to better
>> understand the time line of the File System Switch and the 8th edition network
> f>ile system. 
> 
> 'Introduced with System V Release 3.0, the File System Switch (FSS) architecture introduced a framework under which multiple different filesystem types could coexist in parallel.'
> https://www.oreilly.com/library/view/unix-filesystems-evolution/9780471456759/chap07-sec003.html <https://www.oreilly.com/library/view/unix-filesystems-evolution/9780471456759/chap07-sec003.html>

Thanks for that link!

The SysV R3 source floats around on the web. Its FSS is very different from what is in 8th edition.

In 8th edition the switch has 11 entries (i.e. a file system is an object with 11 virtual methods).
https://github.com/Alhadis/Research-Unix-v8/blob/master/v8/usr/sys/h/conf.h
I have never really studied R3 but at quick inspection the FSS there has 27 (!) entries and seems to be more a sys call switch.

In 10th edition it is still 11 entries, although some refactoring has taken place. Also later work from Research keeps it concise: the 9P protocol from Plan 9 has 14 messages.




[-- Attachment #2: Type: text/html, Size: 2053 bytes --]

Re: [TUHS] 8th Edition timeline

[Paul Ruizendaal]

>> Peter Weinberger, who did it, is at Google; you could ask him
>> directly, as well.
> 
> That is a good idea. If someone has the email address I’d appreciate an off list message.
> 

Email address received off list - many thanks all.

Paul