The Unix Heritage Society mailing list
 help / color / mirror / Atom feed
From: Paul Ruizendaal <>
To: Warner Losh <>
Cc: "" <>
Subject: [TUHS] Re: when did v8 or later get networking?
Date: Sat, 12 Aug 2023 11:06:25 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

> On 12 Aug 2023, at 07:08, Warner Losh <> wrote:
> On Fri, Aug 11, 2023 at 3:05 AM Paul Ruizendaal <> wrote:
> Bill Joy of CSRG concluded that the BBN stack did not perform according to his expectations. Note that CSRG was focused on usage over (thick) ethernet links, and BBN was focused on usage over Arpanet and other wide-area networks (with much lower bandwidth, and higher latency and error rates). He then in 1982 rewrote the stack to match the CSRG environment, changing the design to use software interrupts instead of a kernel thread and optimising the code (e.g. checksumming and fast code paths). It was a matter of debate how new the code was, with the extremes being that it was written from scratch using the spec versus it being mostly copied. Looking at it with a nearly 50 year distance, it seems in between: small bits of surviving SCCS suggest CSRG starting with parts of BBN code followed by rapid, massive modification; the end result is quite different but retained the ‘mbuf’ core data structure and a BBN bug (off-by-one for OOB TCP segments).
> When Kirk McKusick tells  the story, UCB got a beta release (or early access) of the BBN stack. UCB was supposed to add the socket interface to whatever was there. But Bill Joy found it performed terribly (multiple seconds to connect sometimes, single digit kB over 10Mb media, etc). He optimized it to make it perform well. This was a combination of rewriting chunks and tweaking other chunks, which matches your analysis of SCCS. When BBN came back with their new, release ready stack Bill supposedly said something like 'no thanks, we already got one that works way better.' This is why much of the structure of the original BBN stack survived the rewrite: if there wasn't a big issue with them, the design and mechanisms wound up being conserved by this effort. It was too much work to move from mbuf to something else, and too little gain.
> I tried to find a good link, but they are in his BSD history retrospective talks to differing degrees. Sorry I don't have an exact reference.
> Warner


> UCB got a beta release (or early access) of the BBN stack.

This is certainly true. There are four surviving BBN tapes in the CSRG archive, from memory the first was from August 1981 and the last from early 1982. Again from memory, the oldest SCCS entries for the rewrite are from Oct ’81. All of this is on Kirk’s DVD.

Indeed the first tape is early code, written to just interface with Arpanet IMP’s. Things like routing are rudimentary or hard-coded, etc. This is all filled out by the time of the last tape, which also includes a token ring driver (written by Noel Chiappa, if I remember well).

There are two editions of Mike Muuss’ TCP-DIGEST mailing list with posts on performance from Joy and Gurwitz respectively (both from late 1981):

Jonathan Gray sent me a good link of one of Kirk’s talks that addresses this bit of history:


Maybe Kirk was using some hyperbole in that talk. As far as I can tell the main issues were the below:

- BBN coded the checksum routine in C and it compiled badly. Even Joy's hand-optimised version still took some 25% of CPU when traffic maxed out.

- The time-out constants were 2s and 5s. This makes sense for the Arpanet of the time. For local ethernet this was changed to 0.2s and 0.5s.

- The BBN code used at lot of bit fields. This too compiled badly, and was later changed to and/or with #define’d constants

- The BBN code took a very layered approach, often abstracting small bits of functionality into a separate routine. Without compiler inlining and a somewhat slow VAX subroutine mechanism this had a cost. Some of these functions where later changed to #define’s instead of functions.

- Although Kirk singles out replacing the state machine with a big switch statement in the above talk, I’m not sure this was a major performance boost. It is certainly the most visible/recognisable change in the source though. Somehow, this seems to have become core to the debate at the time (ref. the BBN talk at the Summer '84 Usenix conference). Maybe I underestimate the impact of this aspect.

- The TCP management process ran as a kernel thread with normal scheduling. On a loaded machine this meant that it could take seconds for this process to be scheduled again. Changing this to a software interrupt mechanism (somewhat similar to the runrun flag, although Joy appears to have credited VMS for the idea) made things more responsive and avoided context switches.

Of all the above, it would seem to me that only the last point was fundamental to the design. The other things appear relatively easy to fix.


Doug McIlroy wrote: "I recall expressions of surprise (dismay?) at the size of the BSD internet code, but without it we'd have lost our place in the Unix community."

An interesting question is how much of this size is unavoidable. Just counting file lines (i.e. ‘wc -l’), 4.2BSD has some 2200 lines of TCP code (i.e. just the TCP part), the ‘82 BBN implementation 2400 lines, 8th edition had some 2400 lines, and the first version of Plan9 some 2200 lines.

Some parts of TCP could perhaps be simplified (pseudo headers and ‘urgent' segments come to mind), but not much. Maybe this is just the code size that it takes.

URP is smaller, but if I remember well it does not handle out-of-order packets or packet duplicates (both of which do not occur in a (virtual) line switched context).

On the other hand, these sizes compare with 900 lines for the IL protocol in early Plan9.

  parent reply	other threads:[~2023-08-12  9:06 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-08-11  9:05 Paul Ruizendaal
2023-08-12  5:08 ` Warner Losh
2023-08-12  5:41   ` George Michaelson
2023-08-12  9:06   ` Paul Ruizendaal [this message]
2023-08-12 10:29   ` Paul Ruizendaal
2023-08-12 15:20     ` Warner Losh
2023-08-12 15:24       ` Dan Cross
2023-08-12 16:12         ` Paul Ruizendaal
  -- strict thread matches above, loose matches on Subject: below --
2023-08-12 15:05 Noel Chiappa
2023-08-12 18:00 ` Paul Ruizendaal
2023-08-10  1:09 [TUHS] " Larry McVoy
2023-08-10  2:38 ` [TUHS] " segaloco via TUHS
2023-08-10  2:45   ` Warner Losh
2023-08-10  3:17     ` segaloco via TUHS
2023-08-10  3:18     ` Rob Pike
2023-08-10  5:44       ` John Cowan
2023-08-10 12:41 ` Douglas McIlroy
2023-08-10 14:00 ` Jonathan Gray

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).