[TUHS] Re: Origins of the SGS (System Generation Software) and COFF (Common Object File Format)

[segaloco via TUHS <tuhs@tuhs.org>]

For the sake of timelines:

June 1980 - Publication date on the front page of the 3.0 manual in which the utilities are still very much research for PDP-11 and 32V-ish for VAX where distinctions matter.

June 1981 - Publication date on the front page of the 4.1 manual in which the man-pages very much refer to all of this as the "3B-20 object format"

June 1982 - Publication date on the front page of the 5.0 manual by which point these same pages had been edited and extended to describe the "common object file format"

Additions at the 1981 release include dump(1), list(1), and the ld-prefixed library routines for managing these object files.  These likewise persist in 5.0, SysV, and beyond as COFF-related tools.

So this puts the backstop of what would become COFF at at least '81.

- Matt G.

------- Original Message -------
On Thursday, February 23rd, 2023 at 8:49 AM, Paul Winalski <paul.winalski@gmail.com> wrote:


> On 2/22/23, Clem Cole clemc@ccc.com wrote:
> 
> > > - The System V manual has both this ar(1) version as well as the new
> > > COFF-supporting version.
> > > 
> > > Why would ar(1) care?
> > > 
> > > - Not sure if this implies the VAX ar format was expanded to support
> > > the COFF stuff for a little while until they decided on a new one or
> > > what.
> 
> 
> I can't think of any reason why ar(1) would care about the file format
> or internal contents of any of the modules it archives. ar(1) is a
> general archiving tool and can archive anything. It happens that the
> designers of ld(1) decided to use ar(1) to provide searchable object
> file libraries.
> 
> ranlib(1) is a different matter. In order to index global symbols it
> has to understand the object file format(s) of the modules it is
> indexing. ranlib(1) most certainly would have to be taught to
> understand COFF. But not ar(1).
> 
> > > and development software stuff until ELF comes along some time later.
> > 
> > Yep - never quite understood what the push for ELF was over COFF after all
> > the effort to drive COFF down people's throat. Note Microsoft "embraced
> > and extended" COFF as their format -- originally because of Xenix I
> > believe.
> > Someone like Paul W may have some insights on this and that was before
> > the 3B20.
> 
> 
> a.out was, as object file formats go, a throwback to the stone age
> from the get-go. Even the most primitive of IBM's link editors for
> System/360 supported arbitrary naming of object file sections and the
> ability for the programmer to arrange them in whatever order they
> wished. a.out's restriction to three sections (.text, .data, .bss)
> did manage to get the job done, and even (with ZMAGIC) could support
> demand-paged virtual memory, but only just.
> 
> It became pretty clear in the 1980s that an object file format more
> powerful and flexible than a.out was needed. CMU developed their own
> object file format (MACH-O) for their MACH microkernel-based OS. It
> had up to 8 object file sections, and the section properties (e.g.,
> read vs. read/wrkte; executable vs. data) were not tied to the section
> name as in a.out. A big step forward, although still primitive
> compared to the object formats of VAX/VMS and the IBM S/370 OSes.
> Apple MacOS X still uses MACH-O for object files and executables.
> 
> Whatever its origins, what we now know as COFF (Common Object File
> Format) is, as its name implies, intended to be OS- and
> machine-independent. It still has a relatively small number of
> sections, albeit more than MACH-O. When Microsoft developed Windows
> NT, they needed to replace their own MZ executable format with
> something that could support shareable images and they decided to go
> with COFF for both object files and for executables. In typical
> Microsoft embrace-and-extend fashion, their Portable Executable and
> Common Object File Format (PECOFF) is a heavily modified version of
> COFF with lots of MS-specific extensions. When DEC's GEM back end was
> chosen as the optimizer and code generator for Microsoft C/C++ on
> Windows NT for the DEC Alpha chip, I had to add PECOFF support to
> GEM's existing COFF support (which was used by DEC's commercially sold
> compilers for Ultrix). My original idea was to put the PECOFF support
> under conditional compilation (#ifdef PECOFF), but the two formats
> were sufficiently different that I abandoned that Idea, cloned the
> existing COFF module, and then modified that to create a separate
> PECOFF module.
> 
> ELF is far more flexible than either COFF, PECOFF, or MACH-O. Those
> three make a distinction between sections (the bits that eventually
> end up in memory) and the metadata pieces of an object file or
> executable (program headers, symbol table, debug information, etc.).
> In ELF, everything is a section, including the symbol table and the
> tables that direct the program loader in mapping shareable images into
> a process's memory. ELF was originally limited to 64K sections
> (section numbers were unsigned 16-bit), but there is now a scheme for
> 32-bit section numbers. The essentially unlimited number of sections
> is a big boon to languages such as C++, where grouped sections with a
> name-decoration convention provide a convenient way to support sharing
> of class definitions without requiring language-specific tweaks to the
> software development toolset. Contrast this with the Ada
> implementations I'm aware of, which have their own software
> development library systems layered on top of the conventional
> compiler/linker/archiver to insure that program modules are compiled
> and linked in the correct order.
> 
> I don't know what the timeline for the invention of COFF was. It was
> already called COFF and in widespread use by the time I encountered it
> when we added Ultrix support to GEM. I think MACH-O predated COFF;
> it's certainly more primitive than COFF. MACH-O was probably early to
> mid-1980s. OS kernel bloat was a recognized problem at the time and
> microkernel-based OSes were all the rage. At DEC, Dave Cutler wrote a
> microkernel-based OS called VAXeln to replace VAX/VMS for real-time
> applications. A lot of concepts from VAXeln found their way into
> Windows NT when Cutler left DEC for Microsoft.
> 
> -Paul W.