From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HTML_MESSAGE,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 27663 invoked from network); 22 Feb 2023 22:20:52 -0000 Received: from minnie.tuhs.org (50.116.15.146) by inbox.vuxu.org with ESMTPUTF8; 22 Feb 2023 22:20:52 -0000 Received: from minnie.tuhs.org (localhost [IPv6:::1]) by minnie.tuhs.org (Postfix) with ESMTP id A13474223D; Thu, 23 Feb 2023 08:20:45 +1000 (AEST) Received: from mail-vs1-xe30.google.com (mail-vs1-xe30.google.com [IPv6:2607:f8b0:4864:20::e30]) by minnie.tuhs.org (Postfix) with ESMTPS id 6BFBE4223C for ; Thu, 23 Feb 2023 08:20:33 +1000 (AEST) Received: by mail-vs1-xe30.google.com with SMTP id u14so11364257vsp.8 for ; Wed, 22 Feb 2023 14:20:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ccc.com; s=google; t=1677104432; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=3pMFxUiUvo0a+E3I1prdeOx1Ap0Kmy1xFyf8XV0n+78=; b=XeBoRCm5AfmWWJTSYLUTBuE9S0o/ouUevLR6Fe+lXtGQu4nvlQf5bO+CdeDz7zOFgy cH/QP/EoUjBGjzWMTKeYwxlyM/gzWQzcdEpsij/9WXFcv+xM5HfJzPkj2/S4jXBVvs9N K7HtZ7cV8DQdoLfUK9aC3gAl3J1zAHClCVbt4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677104432; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3pMFxUiUvo0a+E3I1prdeOx1Ap0Kmy1xFyf8XV0n+78=; b=Ck7VTFRWmcaRJL4eSf3l5bno9V3Uw9INualKLGTlKCcpkJ7OQvLFsEFVJrXnelH46z SslGqxbvHoe9Fo0mZNm6zhAcUYYLJe2zsmYbY/9A83rFr9S4aLtDb54Cbf4qgShrvQqo AlOf4fVFPt6f+j7ThrXKDbbSAd/dyr+E1aMq4GlDqLYEYBuhrcHPakx6w1ob2ruLX7ZK Xxdrul4kEvCDpLD+rVwtFsuv+jBHcTGe+7XpTYrW+ffQig5cQpgaMpdTWxEniHGIqbxP NdmucxJTtyu/oHhXziQHd0uf7yrERnhIkc1AVmNfvvmaRGULtn1SyeYCdzXruZikKwdE JOJw== X-Gm-Message-State: AO0yUKUi/3nA6Yr4m8bV/LMB6PMjNu5aktTFGYuN67TTaOqZvA9K589d 5Mc9mmG/XvfiFjHDez/7QLnu3a00Qcu+JQaoRIG95ikeY4gURsMj0cg= X-Google-Smtp-Source: AK7set8jKzty6cM9bNKRonxhtMIUBOjKrxpMRBoi4IZXwD3OcuvH7YUaK25maGPQp/jEjGToI4+unhHBY3GcYNU4PkA= X-Received: by 2002:a05:6102:3e8a:b0:3fb:e244:ec1d with SMTP id m10-20020a0561023e8a00b003fbe244ec1dmr1615115vsv.30.1677104431952; Wed, 22 Feb 2023 14:20:31 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Clem Cole Date: Wed, 22 Feb 2023 17:20:05 -0500 Message-ID: To: segaloco Content-Type: multipart/alternative; boundary="000000000000eb0b4d05f5514d4d" Message-ID-Hash: PZO6TC2VVGOTURUYR456CUTE7RYUM4AH X-Message-ID-Hash: PZO6TC2VVGOTURUYR456CUTE7RYUM4AH X-MailFrom: clemc@ccc.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: The Eunuchs Hysterical Society X-Mailman-Version: 3.3.6b1 Precedence: list Subject: [TUHS] Re: Origins of the SGS (System Generation Software) and COFF (Common Object File Format) List-Id: The Unix Heritage Society mailing list Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --000000000000eb0b4d05f5514d4d Content-Type: text/plain; charset="UTF-8" below are some thoughts/hopefully answers to your questions.... On Wed, Feb 22, 2023 at 3:16 PM segaloco via TUHS wrote: > Good day all, figured I'd start a thread on this matter as I'm starting to > piece enough together to articulate the questions arising in my research. > > So based on my analysis of the 3B20S UNIX 4.1 manual I've been working > through, all evidence points to the formalized SGS package and COFF > originating tightly coupled to the 3B-20 line, then growing legs to support > VAX, but never quite absorbing PDP-11 in entirety. That said, there are > bits and pieces of the manual pages for the object format libraries that > suggest there was some providence for PDP-11 in the development of COFF as > well. > > Where this has landed though is a growing curiosity regarding: > > 1. Whether SGS and COFF were tightly coupled to one another from the > outset, with SGS being supported by the general library routines being > developed for the COFF format > > @scj - any enlightenment -- your team in USG must have been part of all that. > > 1. Whether COFF was envisioned as a one-size-fits-all object format > from its inception or started as an experiment in 3B-20 development that > wound up being general enough for other platforms > > That I can not say, but I can say that to the UNIX source licenses (i.e. not the Universities in the Research system or inside of the Bell Systems) - it was used in the "consider it standard" campaign that AT&T marketing in NC was starting to push. This was around the time that PCC2 was coming out to replace the original PCC but I remember getting PCC2 was extra cost. Most of the BSD based kernels (DEC, HP, etc..) were originally using a modified a.out of their own flavor but I think almost all them switched to COFF post the System III license. What I have forgotten, and it may have been a requirement/mixed up in the license. I do remember this was right around when gcc first starts coming out, and they had a tool called robitussin to "cure coffs" as they were using a.out wen they could. > 1. > > > 1. If, prior to this format, there were any other efforts to produce a > unifying binary format and set of development tools, or if COFF was a happy > accident from what were a myriad of different architectural toolset streams > > MIT had a modified a.out format for the NU machine ports - that might have been called b.out. CMU had macho which again was an extended a.out but even more flexible. > > 1. One of the curious things is how VAX for a brief moment did have > its own set of tools and a.out particulars before SGS/COFF. > > Why is that curious - all original Vax development was just using the original PCC stream from V7 (and pre-Judge Green more in a minute). What I don't remember is if PCC2 was COFF when introduced, or COFF can first but I think they were separate things - again someone like scj would be authoritative. The three tools that have to care are the assembler (as), the linker (ld) program loading code in the kernel itself. > For instance, many of the VAX-targeted utilities in 3.0/System III bear > little in common option/manual-wise with the general common SGS utilities > in System V. The "not on PDP-11" pages for various SGS components in System > V much more closely resemble the 3B-20 utilities in 4.1 than any of the non > PDP-11/VAX-only bits in System III. > > Some examples: > > - The VAX assembler in System III contains a -dN option indicating the > number of bytes to set aside for forward/external references for the linker > to fill in. > - The VAX assembler in System V contains among others the -n and -m > options from 4.1 which indicate to disable address optimization and use m4 > respectively > - The System V assembler goes on to also include -R (remove input file > after completion) -r (VAX only, add .data contents to .text instead) and > options -b, -w, and -l to replace the -d1, -d2, and -d4 options indicated > in the previous VAX assembler > - System V further adds a -V to all the SGS software indicating the > version of the software. This is new circa 5.0, absent from the 4.1 manual > like the R, r, b, w, and l options > > > > - The 4.1 manual's singular ar(1) entry still agrees with the System > III version. No arcv(1) is listed, implying the old ar format never made it > to 3B-20 > > Hmm this is confusing old v[456] ar format to new ar format was during Research V6 to Research V7. By the time of any Vax development the old format had pretty much been killed. I'd look at check what PWB 1.0 and 2.0 used. The new ar format was independent of what it was in it. i.e. V7: man 5 ar AR(5) AR(5) NAME ar - archive (library) file format SYNOPSIS #include DESCRIPTION The archive command ar is used to combine several files into one. Archives are used mainly as libraries to be searched by the link-editor ld. A file produced by ar has a magic number at the start, fol- lowed by the constituent files, each preceded by a file header. The magic number and header layout as described in the include file are: #define ARMAG 0177545 struct ar_hdr { char ar_name[14]; long ar_date; char ar_uid; char ar_gid; int ar_mode; long ar_size; }; > > - The System V manual has both this ar(1) version as well as the new > COFF-supporting version. > > Why would ar(1) care? > > - Not sure if this implies the VAX ar format was expanded to support > the COFF stuff for a little while until they decided on a new one or what. > > > > - The System III ld (which is implied to support PDP and VAX) survives > in System V, but is cut down to supporting PDP-11 only > - The COFF-ish ld shows up in 4.1, is then extended to VAX presumably > in the same breath as the other COFF-supporting bits by Sys V, leading to > two copies like many others, PDP-11-specific stuff and then COFF-specific > stuff > > > The picture that starts to form in the context of all of this is, for a > little while in the late 70s/early 80s, the software development > environments for PDP-11, VAX-11, and 3B-20 were interplaying with each > other in often times inconsistent ways. Taking a peek at the 32V manuals, > the VAX tools in System III appear to originate with that project, which > makes sense. If I'm understanding the timeline, COFF starts to emerge from > the 3B-20 project and USG probably decides that's the way to go, a unified > format, but with PDP-11 pretty much out the door support wise already, > there was little reason to apply that to PDP-11 as well, so the PDP-11 > tools get their swan song in System V, original VAX-11 tools from 32V are > likely killed off in 4.x, and the stuff that started with the 3B-20 group > goes on to dominate the object file format > That makes sense - but be careful - the 3B and WE32000 ISA may have been the driver but I would expect that compiler folk in Summit were more in the driver seat. The 3B20 kernel would use what they were getting from the tools team and core kernel team in USG. Remember the politic at the time is Judge Green has unleashed AT&T and they are now allowed to be in the biz, and the sales/marketing folks AT&T was pushing the 3B20 and the WE32000 - so there are big forces behind the scenes that are not obvious/clear. > and development software stuff until ELF comes along some time later. > Yep - never quite understood what the push for ELF was over COFF after all the effort to drive COFF down people's throat. Note Microsoft "embraced and extended" COFF as their format -- originally because of Xenix I believe. Someone like Paul W may have some insights on this and that was before the 3B20. What was the format that the original Xenix used - when it was targeting PDP-11, 68000, x86 and Z8000? Again I'm fuzzy on the details here. But I do remember during the license discussions that would lead to System III, that one of things the Microsoft team was worried about -- IIRC it was Bob Greenberg pushing all that. I lost contact with Bob a few years ago, but if we can find him, I would expect Bob to know what Xenix was doing. And again that negotiation>>starts<< all pre-Judge Green, but finishes up soon afterwards. > > I guess other questions this raises are: > > 1. Were the original VAX tools built with any attention to > compatibility with the PDP-11 bits Ken and Dennis wrote many years prior > (based on some option discrepancies, possibly not?) > > hrmph... folks started with the PDP-11 tools and changed them as needed. I'm not sure compatibility is the right term. They were retargeted nad moved forward by people trying support a new machine they got and did not want run DEC's OS. > > 1. Do the VAX utilities derive from the Interdata 8/32 work or if > there was actually another stream of tools as part of that project? > > I guess I don't understand the question. The original V7 tools were retargeted. When useful features were added, they might be offered/returned to other folks, but remember, Research is not "supporting" UNIX. USG is where things start to think in terms of multiple targets >>before Judge Green<< and then after Judge Green, there was a push to stop using non-AT&T based equipment or chips in the Bell System and make what Western Electric was selling be attractive [which sometimes was a little bit of putting lipstick on porcine as it were]. For instance, Rob and Barts's original JERQ is 68000 based, but by the time it becomes a product as 5620 it has to be refactored as a WE32000. > > 1. Was there any interplay between the existing tool streams (original > PDP-11, 32V's VAX utilities, possibly Interdata 8/32) and the eventual > COFF/SGS stuff, or was the latter pretty well siloed in 3B-20 land until > deployment with 4.1? > > I think you are putting too much on the 3B program itself. The 3B was the task at hand at the time and a solid opportunity to bring to bear business choices being made. You need to look at the greater business to understand a lot of the choices. A lot of things were happening in parallel in the market that had other impacts on technology and how it was delivered -- the 3B program was the "technology train" leaving the station that some of them got attached to/delivered using. But, I as I said to you when we chatted, you really can not underestimate what was happening (or not happening) as AT&T changed its business focus - pre/post-Judge Green. It was a large company with lots of different spheres of interest (read - different executives), each being measured with different things that they might value. --000000000000eb0b4d05f5514d4d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
below are some= thoughts/hopefully answers to your questions....

On Wed, Feb 2= 2, 2023 at 3:16 PM segaloco via TUHS <t= uhs@tuhs.org> wrote:
Good day all, figured I'd start a thread on this matte= r as I'm starting to piece enough together to articulate the questions = arising in my research.

So based on my analysis of the 3B2= 0S UNIX 4.1 manual I've been working through, all evidence points to th= e formalized SGS package and COFF originating tightly coupled to the 3B-20 = line, then growing legs to support VAX, but never quite absorbing PDP-11 in= entirety. That said, there are bits and pieces of the manual pages for the= object format libraries that suggest there was some providence for PDP-11 = in the development of COFF as well.

Where this has landed = though is a growing curiosity regarding:
  1. Whether SG= S and COFF were tightly coupled to one another from the outset, with SGS be= ing supported by the general library routines being developed for the COFF = format
= @scj - any enlightenment=C2=A0-- your team in USG must have been part of al= l that.=C2=A0
  1. Whether COFF was envisioned as a one-size-fits-all object = format from its inception or started as an experiment in 3B-20 development = that wound up being general enough for other platforms
=
That I can not say, but= I can say that to the UNIX source licenses (i.e. not the Universities in t= he Research system or inside of the Bell Systems) - it was used in the &quo= t;consider it standard" campaign=C2=A0that AT&T marketing= =C2=A0in NC was starting to=C2=A0push.=C2=A0 This was around= the time that PCC2 was coming out to replace the original PCC but I rememb= er getting PCC2 was extra cost.=C2=A0=C2=A0

Most of the BSD based kernels (DEC, HP, etc..) were originally using a m= odified a.out of their own flavor but I think almost all them switched to C= OFF post the System III license.=C2=A0 =C2=A0What I have forgotten, and it = may have been a requirement/mixed up in the license.=C2=A0=C2=A0

<= font color=3D"#0000ff">I do remember this was right around when gcc first s= tarts coming out, and they had a tool called robitussin=C2=A0to "cure = coffs" as they were using a.out wen they could.

  1. If, prior to this fo= rmat, there were any other efforts to produce a unifying binary format and = set of development tools, or if COFF was a happy accident from what were a = myriad of different architectural toolset streams
MIT had a modified a.out format for the N= U machine ports=C2=A0- that might have been called b.out.=C2=A0 = =C2=A0 =C2=A0
CMU had= macho which again was an extended a.out but even more flexible.
    =
  1. = One of the curious things is how VAX for a brief moment did have its own se= t of tools and a.out particulars before SGS/COFF.
Why is that curious - all original Vax dev= elopment was just using the original PCC stream from V7=C2=A0 (and pre-Judg= e=C2=A0Green more in a minute).

=
What I don't remember=C2=A0is i= f PCC2 was COFF when introduced, or COFF can first but I think they were se= parate things - again someone like scj would be authoritative.
=

The = three tools that have to care are the assembler (as), the linker (ld) progr= am loading code in the kernel itself.

=C2=A0

=C2=A0
For instance, many of= the VAX-targeted utilities in 3.0/System III bear little in common option/= manual-wise with the general common SGS utilities in System V. The "no= t on PDP-11" pages for various SGS components in System V much more cl= osely resemble the 3B-20 utilities in 4.1 than any of the non PDP-11/VAX-on= ly bits in System III.

Some examples:
  • = The VAX assembler in System III contains a -dN option indicating the number= of bytes to set aside for forward/external references for the linker to fi= ll in.
  • The VAX assembler in System V contains among ot= hers the -n and -m options from 4.1 which indicate to disable address optim= ization and use m4 respectively
  • The System V assembler= goes on to also include -R (remove input file after completion) -r (VAX on= ly, add .data contents to .text instead) and options -b, -w, and -l to repl= ace the -d1, -d2, and -d4 options indicated in the previous VAX assembler
  • System V further adds a -V to all the SGS software indi= cating the version of the software. This is new circa 5.0, absent from the = 4.1 manual like the R, r, b, w, and l options

  • The 4.1 manual's singular ar(1) entry still agrees= with the System III version. No arcv(1) is listed, implying the old ar for= mat never made it to 3B-20
Hmm this is confusing old v[456] ar format to new = ar format was during Research V6 to Research V7.=C2=A0 By the time of any V= ax development the old format had pretty much been killed. I'd look at = check=C2=A0what PWB 1.0 and 2.0 used. The new ar format was independent of = what it was in it.

=
i.e. V7: man 5 ar=C2=A0
  AR(5)                                                       AR(5)

     NAME
          ar - archive (library) file format

     SYNOPSIS
          #include <ar.h>

     DESCRIPTION
          The archive command ar is used to combine several files into
          one.  Archives are used mainly as libraries to be searched
          by the link-editor ld.

          A file produced by ar has a magic number at the start, fol-
          lowed by the constituent files, each preceded by a file
          header.  The magic number and header layout as described in
          the include file are:

<= div>
#define ARMAG 0177545
struct ar_hdr {
char ar_name[14];
long ar_date;
=
char ar_uid;
char ar_gid;
int ar_mode;
long ar_s= ize;
<= div>
};



=C2=A0
  • The = System V manual has both this ar(1) version as well as the new COFF-support= ing version.
Why would ar(1) care?

=C2=A0
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex">
  • Not su= re if this implies the VAX ar format was expanded to support the COFF stuff= for a little while until they decided on a new one or what.

    • The System III ld (which is implied to = support PDP and VAX) survives in System V, but is cut down to supporting PD= P-11 only
    • The COFF-ish ld shows up in 4.1, is then ext= ended to VAX presumably in the same breath as the other COFF-supporting bit= s by Sys V, leading to two copies like many others, PDP-11-specific stuff a= nd then COFF-specific stuff

    The picture= that starts to form in the context of all of this is, for a little while i= n the late 70s/early 80s, the software development environments for PDP-11,= VAX-11, and 3B-20 were interplaying with each other in often times inconsi= stent ways. Taking a peek at the 32V manuals, the VAX tools in System III a= ppear to originate with that project, which makes sense. If I'm underst= anding the timeline, COFF starts to emerge from the 3B-20 project and USG p= robably decides that's the way to go, a unified format, but with PDP-11= pretty much out the door support wise already, there was little reason to = apply that to PDP-11 as well, so the PDP-11 tools get their swan song in Sy= stem V, original VAX-11 tools from 32V are likely killed off in 4.x, and th= e stuff that started with the 3B-20 group goes on to dominate the object fi= le format
That makes sense - but be careful - the 3B and WE32000 ISA may have bee= n the driver but I would expect that compiler folk in Summit were more in t= he driver seat.=C2=A0 =C2=A0The 3B20 kernel would use what they were gettin= g from the tools team and core kernel team in USG.=C2=A0

Remember t= he politic at the time is Judge Green has unleashed AT&T and they are n= ow allowed to be in the biz, and the=C2=A0 sales/marketing folks AT&T w= as pushing the 3B20 and the WE32000 - so there=C2=A0are big forces behind t= he scenes that are not obvious/clear.

=C2=A0
and dev= elopment software stuff until ELF comes along some time later.
<= /div>
Yep - never quite = understood what the push for ELF was over COFF=C2=A0after all the effort to= drive=C2=A0COFF down people's throat.=C2=A0 =C2=A0Note Microsoft "= ;embraced and extended" COFF as their format -- originally because of = Xenix I believe.=C2=A0=C2=A0 Some= one=C2=A0like Paul W may have some insights on this and that was before the= 3B20.=C2=A0

What was the format that the= original Xenix used - when it was targeting PDP-11, 68000, x86 and Z8000?= =C2=A0=C2=A0Again I'm fuzzy on the details here. But I do = remember during the license discussions that would lead to System III, that= one=C2=A0of things the Microsoft team was worried about -- IIRC it w= as Bob Greenberg pushing all that.=C2=A0 I lost contact with= Bob a few years ago, but if we can find him, I would expect Bob to know wh= at Xenix was doing.=C2=A0 And again that negotiation>>starts<< = all pre-Judge=C2=A0Green, but finishes up soon afterwards.

I guess other questions this raises are:
  1. W= ere the original VAX tools built with any attention to compatibility with t= he PDP-11 bits Ken and Dennis wrote many years prior (based on some option = discrepancies, possibly not?)
<= /blockquote>
hrmph... folks started with th= e PDP-11 tools and changed them as needed. I'm not sure compatibility= =C2=A0is the right term.=C2=A0 They were retargeted nad moved forward by pe= ople trying support a new machine they got and did not want run DEC's O= S.
  1. = Do the VAX utilities derive from the Interdata 8/32 work or if there = was actually another stream of tools as part of that project?
I g= uess I don't understand the question.=C2=A0 The original V7 tools were = retargeted.=C2=A0 When useful features were added, they might be offered/re= turned=C2=A0to other folks, but remember, Research is not "supporting&= quot; UNIX.=C2=A0 USG is where things start to think in terms of multiple t= argets >>before Judge Green<< and then after Judge Green, there= was a push to stop using non-AT&T based equipment or chips in the Bell= System and make what Western=C2=A0Electric was selling be attractive [which sometimes was a li= ttle bit of putting lipstick on porcine as it were].=C2= =A0 For instance,=C2=A0Rob= and Barts's original JERQ is= 68000 based, but by the time it becomes a product as 5620 it has to be ref= actored as a WE32000.

=C2=A0
  1. Was there any interplay betwee= n the existing tool streams (original PDP-11, 32V's VAX utilities, poss= ibly Interdata 8/32) and the eventual COFF/SGS stuff, or was the latter pre= tty well siloed in 3B-20 land until deployment with 4.1?
I think= you are putting too much on the 3B program itself.=C2=A0 The 3B was the ta= sk at hand at the time and a solid opportunity to bring to bear business ch= oices being made.=C2=A0 You need to look at the greater business to underst= and a lot of the choices.=C2=A0 A lot of things were happening in parallel = in the market that had other impacts on technology and=C2=A0how it was deli= vered -- the 3B program was the "technology train" leaving the st= ation that some of them got attached to/delivered using.=C2=A0=C2=A0=

= But, I as I said to you when we chatted, you re= ally can not underestimate=C2=A0what was happening (or not happening) as AT= &T changed its business focus - pre/post-Judge Green. It was a large co= mpany with lots of different spheres of interest (read - different executiv= es), each being measured with different things that they might value.=C2=A0=
--000000000000eb0b4d05f5514d4d--