* [TUHS] shared objects in Unix @ 2018-03-29 21:37 Paul Winalski 2018-03-29 22:01 ` Henry Bent ` (3 more replies) 0 siblings, 4 replies; 22+ messages in thread From: Paul Winalski @ 2018-03-29 21:37 UTC (permalink / raw) The recent discussion of long-lived applications, and backwards compatibility in Unix, got me thinking about the history of shared objects. My own experience with Linux and MacOS is that statically-linked applications tend to continue working from release to release, but shared objects provided by the OS tend not to be backwards compatible, and one often has to carry around with your application the exact C runtime and other shared objects your program was linked against. This is in big contrast to shared libraries on VMS, where great care is taken to maintain strict backward compatibility release to release. What is the history of shared objects on Unix? When did they first appear, and with what object/executable file format? The a.out ZMAGIC format doesn't seem to support them. I don't recall if COFF does. MACH-O, at least the MacOS dialect of it, supports dynamic libraries. ELF supports them. Also, when was symbol preemption invented? Traditional shared library designs such as in IBM System/370, VMS, and Windows NT doesn't have it. As one who worked on optimizations in compilers, I came to hate symbol preemption because it prohibits many useful optimizations. ELF does provide a way to turn it off, but it's on by default--you have to explicitly declare symbols as protected or hidden via source language pragmas to get rid of it. -Paul W. ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 21:37 [TUHS] shared objects in Unix Paul Winalski @ 2018-03-29 22:01 ` Henry Bent 2018-03-29 22:26 ` Warner Losh ` (2 subsequent siblings) 3 siblings, 0 replies; 22+ messages in thread From: Henry Bent @ 2018-03-29 22:01 UTC (permalink / raw) On 29 March 2018 at 17:37, Paul Winalski <paul.winalski at gmail.com> wrote: > > What is the history of shared objects on Unix? When did they first > appear, and with what object/executable file format? The a.out ZMAGIC > format doesn't seem to support them. I don't recall if COFF does. > IRIX 4 had COFF shared libraries, barely. The OS included libc_s, libgl_s, and maybe one or two other things. They were very difficult and time-consuming to create, and if you had external dependencies then things were even worse. -Henry -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/4eeeddb3/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 21:37 [TUHS] shared objects in Unix Paul Winalski 2018-03-29 22:01 ` Henry Bent @ 2018-03-29 22:26 ` Warner Losh 2018-03-29 23:11 ` Dave Horsfall 2018-03-30 0:40 ` Clem Cole 3 siblings, 0 replies; 22+ messages in thread From: Warner Losh @ 2018-03-29 22:26 UTC (permalink / raw) On Thu, Mar 29, 2018 at 3:37 PM, Paul Winalski <paul.winalski at gmail.com> wrote: > > What is the history of shared objects on Unix? When did they first > appear, and with what object/executable file format? The a.out ZMAGIC > format doesn't seem to support them. I don't recall if COFF does. > MACH-O, at least the MacOS dialect of it, supports dynamic libraries. > ELF supports them. > Both FreeBSD and Linux supported shared libraries for a.out, though I can't recall which of the *MAGIC formats they were. The Linux ones had fixed load addresses, while the FreeBSD ones allowed any load address. Each of these approaches has pros and cons, but both were tossed away in favor of ELF just as soon as ELF was stable. Though, FreeBSD still has an a.out run time linker in the tree. I wouldn't think it was still in use, but the maintainer still fixes a bug in it every 9-24 months that some user has reported... Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/53060dae/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 21:37 [TUHS] shared objects in Unix Paul Winalski 2018-03-29 22:01 ` Henry Bent 2018-03-29 22:26 ` Warner Losh @ 2018-03-29 23:11 ` Dave Horsfall 2018-03-29 23:22 ` Warner Losh ` (2 more replies) 2018-03-30 0:40 ` Clem Cole 3 siblings, 3 replies; 22+ messages in thread From: Dave Horsfall @ 2018-03-29 23:11 UTC (permalink / raw) On Thu, 29 Mar 2018, Paul Winalski wrote: [...] > What is the history of shared objects on Unix? When did they first > appear, and with what object/executable file format? The a.out ZMAGIC > format doesn't seem to support them. I don't recall if COFF does. > MACH-O, at least the MacOS dialect of it, supports dynamic libraries. > ELF supports them. I first saw 'em when they appeared in SunOS (can't remember which release) and thought they were wonderful, along with loadable drivers. -- Dave Horsfall DTM (VK2KFU) "Those who don't understand security will suffer." ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 23:11 ` Dave Horsfall @ 2018-03-29 23:22 ` Warner Losh 2018-03-29 23:24 ` Larry McVoy 2018-03-30 1:20 ` Lyndon Nerenberg 2 siblings, 0 replies; 22+ messages in thread From: Warner Losh @ 2018-03-29 23:22 UTC (permalink / raw) On Mar 29, 2018 5:12 PM, "Dave Horsfall" <dave at horsfall.org> wrote: On Thu, 29 Mar 2018, Paul Winalski wrote: [...] What is the history of shared objects on Unix? When did they first appear, > and with what object/executable file format? The a.out ZMAGIC format > doesn't seem to support them. I don't recall if COFF does. MACH-O, at > least the MacOS dialect of it, supports dynamic libraries. ELF supports > them. > I first saw 'em when they appeared in SunOS (can't remember which release) and thought they were wonderful, along with loadable drivers. Shared libraries were 4.0 due to the new mmap call. These were a.out based I'm pretty sure. Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/f13c9df8/attachment-0001.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 23:11 ` Dave Horsfall 2018-03-29 23:22 ` Warner Losh @ 2018-03-29 23:24 ` Larry McVoy 2018-03-30 0:22 ` Clem Cole 2018-03-30 1:20 ` Lyndon Nerenberg 2 siblings, 1 reply; 22+ messages in thread From: Larry McVoy @ 2018-03-29 23:24 UTC (permalink / raw) On Fri, Mar 30, 2018 at 10:11:42AM +1100, Dave Horsfall wrote: > On Thu, 29 Mar 2018, Paul Winalski wrote: > > [...] > > >What is the history of shared objects on Unix? When did they first > >appear, and with what object/executable file format? The a.out ZMAGIC > >format doesn't seem to support them. I don't recall if COFF does. MACH-O, > >at least the MacOS dialect of it, supports dynamic libraries. ELF supports > >them. > > I first saw 'em when they appeared in SunOS (can't remember which release) > and thought they were wonderful, along with loadable drivers. Warner and I have been going back and forth about this. We're both pretty sure that shared libs were part of the 4.0 release (that was the release that had the VM system rewrite by Joe Moran (mojo at sun.com) with mmap() as invisioned by Bill Joy while still at Berkeley. I've got a number of papers about it: http://mcvoy.com/lm/papers/SunOS.shlib.pdf - that's the shared lib paper http://mcvoy.com/lm/papers/SunOS.vm_arch.pdf - that's the VM architectue paper http://mcvoy.com/lm/papers/SunOS.vm_impl.pdf - that's the VM implementation paper There's other stuff there too if you are bored. SunOS.smoosh.pdf is the basic idea that is the under pinnings of every distributed source management system, SunOS.tfs.pdf is Dave Hendrick's copy on write file system writeup, SunOS.ufs_clustering.pdf is the work I did in UFS to get platter speed perf out of UFS, etc. Most of that stuff is Sun stuff though there is some other random bits. If you are curious about any of it I can go into detain off (or on) list. --lm ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 23:24 ` Larry McVoy @ 2018-03-30 0:22 ` Clem Cole 2018-03-30 1:46 ` Larry McVoy 0 siblings, 1 reply; 22+ messages in thread From: Clem Cole @ 2018-03-30 0:22 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 2270 bytes --] Paul, The point is that there were a number of different shared library implementations for UNIX over the years. That was one of the 'knocks' when comparing VMS to 4.1BSD in the early 80s. VMS had shared libraries from the start. I'm pretty sure the first unix to support shared libraries was CMU's Mach using its modified about called macho, which lives today in Mac OSX. BSD4.2 had started to implement it, but it was incomplete so folks like Sun and Masscomp each did their own scheme, both based on modified a.out. A big problem was every vendor messed with a.out in a different way -- so Sun's, Masscomp and Apollo's versions were all a little different and you a linker guy, you know that a.out was not a great format for same. With System V.3, AT&T introduced was an a new file format, called the Common Object File Format - *a.k.a.* COFF. SVR3 supported shared libs. In fairness, COFF was a huge improvement over a.out, but it was done when AT&T was in its 'consider it standard' time and trying to force its will and wanted licensing fees. Let's say that failed for non-technical reasons. Unfortunately, it lead to more confusion and we ended up a number of different COFF-almost, sort-of, extensions. IIRC, IBM went with a modified COFF, but again we were in a cold war of who could do what. I remember that the time, the Gnu guys wrote tool called 'robitussin' - which 'cured COFFs.' With, AT&T's SVR4 release the world was introduced to the 'Extended Linker Format' - ELF, which fixed a number of issues with COFF, the primary one being that it could loaded images faster and you could page from directly, which neither a.out nor COFF could easily. Again IIRC, SVR4's linker could handled AT&T's COFF files. I have never known the legals on it, but some how the details of ELF did become public and somebody reimplemented the GNU compilers to use it and AT&T for whatever reason did not complain (maybe the had their hands full at time with BSDi case). Anyway, eventually both Linux and FreeBSD switched to that version of Gnu and its pretty much been stable since. ᐧ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/455b07c7/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 0:22 ` Clem Cole @ 2018-03-30 1:46 ` Larry McVoy 2018-03-30 4:28 ` Clem cole 0 siblings, 1 reply; 22+ messages in thread From: Larry McVoy @ 2018-03-30 1:46 UTC (permalink / raw) On Thu, Mar 29, 2018 at 08:22:14PM -0400, Clem Cole wrote: > I'm pretty sure the first unix to support shared libraries was > CMU's Mach using its modified about called macho, which lives today in Mac > OSX. Uh, you sure about that? http://cs.cmu.edu/afs/cs.cmu.edu/project/mach/public/doc/published/mapfiles87.ps is as close as I can find, and that's talking about stuff that was long after Sun's shared libraries. There may have been earlier stuff but the approach laid out in http://www.mcvoy.com/lm/papers/SunOS.shlib.pdf is pretty much the shared library world as we know it today so far as I know. I remember the world before that, I lived in it, and shared libraries were not a working thing in my memory. Maybe on VMS, I didn't program much on VMS, but on any Unix I could get my hands on, Sun was the first to have working shared libraries. CMU's Mach, mem, I am by no means a fan (I bought into the hype, read all the papers, when I finally got to see the code, wow. NOTHING like Sun's VM system, I mean, nothing. It claimed to be the same sort of thing, it was an ugly mess and it still is. Sun's VM system was a thing of beauty, you could read the code and figure out the architecture from the code. I'd challenge anyone to do that from the Mach code). But maybe it had shared libs before SunOS but who was using that code? So far as I know the first time the Mach code was in a commercial product was Next. Their first release was October 1988, SunOS 4.0 was Dec 1988 and was a far far far more mature release. Here's a way to put it into perspective. In the mid 80's, and maybe before, every single open source project's makefile just worked on a Sun. I don't ever remember seeing a Makefile that just worked on a Next (and I don't know of any other Mach based platform until Apple many years later). ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 1:46 ` Larry McVoy @ 2018-03-30 4:28 ` Clem cole 2018-03-30 20:52 ` Jon Forrest 0 siblings, 1 reply; 22+ messages in thread From: Clem cole @ 2018-03-30 4:28 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 4264 bytes --] You might be right i don’t exactly remember the dates of each release. that said I do remember Mach ran on a number of systems before Next, but they were not commercial. It was similar to BSD where CMU ported to that hardware - I’ve forgotten the name of the guys that spun out of DG that had an early MP based system that used reflected memory- but I remember CMU had very early version of Mach running there that was based on the 4.1mash up like the original Vax code from CMU that they started. The point is that early Mach was working before 4.2 way final and released to the DARPA community not much later. CRSG was doing Val support for DARPA and CMU was more researchy (which was a point of contention in a lot of fronts that I’ll not scratch open any further). I was not commenting on the implementation of the goodness of the CMU or CSRG code by the wsy. Sun could have had the first commercial version of shared libs that worked well, although IBM might have been around the same time as Charlie pointed. What had Paul asked when shared libs came to being for Unix. I was trying to say it did not come from any one point. I do believe Mach and it’s parent Accent (which was not Unix) was the first time any Unix had it but like BSD was a DARPA funded project and not commercial. But a lot of different places were working on the idea as it was commonly held to be an issue with Unix. Also, joy / BSD 4.2 was heavily influenced by Accent (and RIG )and the Mach memory system would eventually go back into BSD (4.3 IIRC) - which we have talked about before wrt to sockets and Accent/Mach’s port concept. FWIW If I'm remembering the sequence right I believe Mach 2.5 was quickly created as a update to the original release after joy et al released the final 4.2. 4.1c was 83 as I was leaving UCB and I brought it with me. But I think we had an early 4.1 based Mach tape at Masscomp not too much later ??a year maybe?? And one of the reasons we were interested in it was the shared library code because the exVMS at Masscomp all felt that was a deficiency of Unix and at the time the only ‘open source’ in a HLL if you will that had it was Mach. Sent from my PDP-7 Running UNIX V0 expect things to be almost but not quite. > On Mar 29, 2018, at 9:46 PM, Larry McVoy <lm at mcvoy.com> wrote: > >> On Thu, Mar 29, 2018 at 08:22:14PM -0400, Clem Cole wrote: >> I'm pretty sure the first unix to support shared libraries was >> CMU's Mach using its modified about called macho, which lives today in Mac >> OSX. > > Uh, you sure about that? > > http://cs.cmu.edu/afs/cs.cmu.edu/project/mach/public/doc/published/mapfiles87.ps > > is as close as I can find, and that's talking about stuff that was long after > Sun's shared libraries. > > There may have been earlier stuff but the approach laid out in > > http://www.mcvoy.com/lm/papers/SunOS.shlib.pdf > > is pretty much the shared library world as we know it today so far as I > know. > > I remember the world before that, I lived in it, and shared libraries were > not a working thing in my memory. Maybe on VMS, I didn't program much on > VMS, but on any Unix I could get my hands on, Sun was the first to have > working shared libraries. > > CMU's Mach, mem, I am by no means a fan (I bought into the hype, read > all the papers, when I finally got to see the code, wow. NOTHING like Sun's > VM system, I mean, nothing. It claimed to be the same sort of thing, it was > an ugly mess and it still is. Sun's VM system was a thing of beauty, you > could read the code and figure out the architecture from the code. I'd > challenge anyone to do that from the Mach code). > > But maybe it had shared libs before SunOS but who was using that code? > So far as I know the first time the Mach code was in a commercial product > was Next. Their first release was October 1988, SunOS 4.0 was Dec 1988 > and was a far far far more mature release. > > Here's a way to put it into perspective. In the mid 80's, and maybe > before, every single open source project's makefile just worked on > a Sun. I don't ever remember seeing a Makefile that just worked on > a Next (and I don't know of any other Mach based platform until Apple > many years later). ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 4:28 ` Clem cole @ 2018-03-30 20:52 ` Jon Forrest 2018-03-30 22:42 ` Clem Cole 0 siblings, 1 reply; 22+ messages in thread From: Jon Forrest @ 2018-03-30 20:52 UTC (permalink / raw) [Larry McVoy said ...] >> CMU's Mach, mem, I am by no means a fan (I bought into the hype, read >> all the papers, when I finally got to see the code, wow. NOTHING like Sun's >> VM system, I mean, nothing. It claimed to be the same sort of thing, it was >> an ugly mess and it still is. This is typical of university research projects. To those of us who worked in the Postgres research group at UC Berkeley, one of the great mysteries of the world is how the PostgreSQL community was able to take the research Postgres code and make it into the production quality database it is now. Most research projects suffer because the goal of the people who work on it is to hack on it to get their research done, so that they can get their MS/PhD and then get the hell out. Code quality is rarely a major concern. Postgres (and Ingres) benefited from having a Chief Programmer who attempted to minimize this problem. Jon Forrest ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 20:52 ` Jon Forrest @ 2018-03-30 22:42 ` Clem Cole 2018-03-30 23:29 ` Paul Winalski 2018-04-03 15:49 ` Derek Fawcus 0 siblings, 2 replies; 22+ messages in thread From: Clem Cole @ 2018-03-30 22:42 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 5667 bytes --] On Fri, Mar 30, 2018 at 4:52 PM, Jon Forrest <nobozo at gmail.com> wrote: > [Larry McVoy said ...] > >> CMU's Mach, mem, >>> ... >>> NOTHING like Sun's >>> VM system, I mean, nothing. >>> >> > ... > > Most research projects suffer because the goal of the people who > > work on it is to hack on it to get their research done, Two things... 1.) I think we are in complete agreement wrt to code quality - and I think LM is probably correct that Sun had the first real quality implementation. 2.) Paul's question was a little different and I was able to chat with him about it at lunch today. Until a year or so ago, Paul was a compiler/linker guy and was interested less in the OS extension to support shared libraries, but more when did different tool chains start to support it. * i.e.* How did we get from a.out to ELF. The key is that make shared libraries real for the tool chain, you need OS support for shared memories. It turns out that Joy's BSD Unix took its shared memory API (mmap) from Tenex's 'PMAP JSYS' (which BTW, the '72 Tenex CACM paper calls out shared libraries modelled after Multics - I was trying to google the API specific and failed - I think its darned near 1x1). I'm also pretty sure that PMAP influenced the VMS shared memory call that Cutler would build in the mid 70s. Not only that, In Rashid's 81 Accent paper, he too referenced Tenex's PMAP too (more in a minute). Also, please remember, that BSD's mmap was not the first shared memory call for any UNIX. Horton & Johnson can correct me, but I think Dale's team in Columbus had a shared memory in a PDP-11 ??PWB 1.0?? style system - that would eventually begat the System V shmem call (i've got a hard copy of it somewhere in my archives). What I do not remember, and this is where someone like Steve might know, was could COFF use the Columbus shmem to do shared libs? Certainly by that time, shared libraries for UNIX was being heavily discussed both in and out of AT&T. I remember that either V.2 or V.3 had a shared lib scheme that put the libraries at a static address as the System V kernel support was not very flexible. But to Paul's question, was that AT&T had started to put shared libs into their flavor around the same time? *i.e. *when did Summit start to support it. I was trying to get an account of which versions of which OS distributions/releases had some sorts of kernel extensions that allowed shared libraries and that also supported tool chains that could provide them -- *i.e.* what Paul was asking about. I was not commenting on who's kernel to support same was good or bad. I did comment on the file format. COFF was a dead end because of the way Summit's management introduced it to larger Unix community, even if it was 'better' and solved some of the earlier issues with a.out. Macho was already there for instance, and it was an 'extensible' version of a.out and had some level of support for architecture of the binary contained within. How v6/v7 a.out got extended to support shared libs, had varied widely by different manufacturers tool chains. Originally, most manufacturers still called it a.out and just hacked on their own version of the toolchain. CMU gave it a different name (macho) and IIRC made the kernel handle both *i.e. *still supported the original BSD a.out but you were limited in what could be done with it to only the 4.1BSD api. Remember, Mach 1.0, reacting to Accent' failure, was BSD4.1++ so they wanted to be able to run BSD binaries, at least on Vaxen. In fact the way you installed Mach1.0 was build a new kernel boot it on your 4.1BSD system. Binary compatibility was sort of new concept for the University community --- in fact, around that time at UCB I remember Joy once saying he thought it was good to force people to recompile as it ensured the code was not stale. So moving forward, AT&T's COFF had tried to repair the many different a.outs, but AT&T made other major non-technical missteps so only firms that were late in the Unix world bothered with it. And even after trying to be 'common', it too, got extended in N different incompatible (uncommon) ways. As I said, I am not sure how ELF really became the universal format, other than timing. AT&T for whatever reason did not have a fit when it was reimplemented by Gnu - but by the Gnu guys doing that, and the FOSS community switching to that version of the compiler, ELF did win in the long run. And fortunately, the Gnu version of ELF and the SVR4 version seems/appears to be were pretty close [as far as I know they matched, but if someone knows otherwise, I defer to them]. As for the kernel stuff, my other point is that >>idea<< of mmap/pmap was kicking around the Unix community by numerous folks hacking on the kernel for a longer time then when it was finally implemented. LM reminds us that you need something for memory sharing to implement shared libraries. But memory sharing added to Unix before shared libraries was added to the tool chains. I do find it interesting the UCB mmap API call pretty much was all of the Unix kernel implementation agreed to be the memory sharing API and that it happens to be pretty much the same call as Tenex's PMAP. As Paul mentioned at lunch today for him, VMS, Tenex/TOP-20 and eventually Unix it was pretty much the same call for him implementing the user space linker/loader calls, which is part of why he was asking when did the tools start to support it all. Clem ᐧ ᐧ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180330/2341c230/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 22:42 ` Clem Cole @ 2018-03-30 23:29 ` Paul Winalski 2018-04-03 15:49 ` Derek Fawcus 1 sibling, 0 replies; 22+ messages in thread From: Paul Winalski @ 2018-03-30 23:29 UTC (permalink / raw) Regarding the various object/program image formats on Unix: a.out ZMAGIC had three sections (.text, .data, BSS) and .text was read-only and therefore shareable between processes running the same image. And if .text and .data started on a page boundary, you could do demand paging of them from the a.out file on disk (you'd of course need to do copy-on-write of .data to space in a page file). I can see how, with mmap, one could map multiple ZMAGIC images to the same address space, thus implementing shared objects, but there isn't anything in ZMAGIC to direct the runtime loader as to which images need to be so mapped or where to map them. MACH-O was a big advance over a.out in that it was extensible--you could have up to 16 sections. MacOS-X indeed uses some of the extra sections to implement its shared memory scheme. COFF was another step forward. One now had a lot more sections to play with. But a lot of the meta-data needed by the runtime loader to do shared image binding was in auxiliary data structures outside such as the optional header (which in practice is anything but optional). And, as Clem pointed out, it suffered from a dialect problem. Microsoft adopted COFF as the object and image format for Windows NT. But as MS does so often, they took a "embrace and extend" approach to it. When I had to implement object file writing support in DEC's GEM back end for Microsoft PECOFF, I found it significantly different from Tru64 Unix's COFF. I found myself putting so many "if (is_pecoff)" conditionals in the code that I gave up on that and wrote a separate module for PECOFF (just as the VMS OBJ support had its own module). The two COFF-based shared object designs I'm familiar with (Tru64 Unix and Windows NT) both hung the data structures for shared object loading off of the optional header. ELF is the cleanest and the easiest to work with, from a compiler writer's point of view. Everything is a section. The one mistake they made was using a 16-bit field for the section number, thus limiting each file to 64K sections. The grouped communal sections for C++ can blow through that limit quite easily. A hack was later added to ELF to support 32-bit section numbers. It's not as clean as it would have been if section numbers had been 32 bits from the get-go, but it does mean that only modules that need a grossly large number of sections incur the file bloat from the larger section numbers. ELF is nice enough that when VMS did their port to Itanium, they decided to use ELF rather than try to add Itanium's extensive set of relocations to the OBJ format in use on Alpha. -Paul W. ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 22:42 ` Clem Cole 2018-03-30 23:29 ` Paul Winalski @ 2018-04-03 15:49 ` Derek Fawcus 1 sibling, 0 replies; 22+ messages in thread From: Derek Fawcus @ 2018-04-03 15:49 UTC (permalink / raw) On Fri, Mar 30, 2018 at 06:42:09PM -0400, Clem Cole wrote: > > And fortunately, the Gnu version of ELF and the SVR4 version > seems/appears to be were pretty close [as far as I know they matched, but > if someone knows otherwise, I defer to them]. Well, there was a subtle difference between the Solaris SPARC version and the Linux i386 version which we ran in to when porting a program. I can't quite recall, something to do with symbol visibility for symbols defined in the main executable vs those in a shared object. DF ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 23:11 ` Dave Horsfall 2018-03-29 23:22 ` Warner Losh 2018-03-29 23:24 ` Larry McVoy @ 2018-03-30 1:20 ` Lyndon Nerenberg 2 siblings, 0 replies; 22+ messages in thread From: Lyndon Nerenberg @ 2018-03-30 1:20 UTC (permalink / raw) > I first saw 'em when they appeared in SunOS (can't remember which release) > and thought they were wonderful, along with loadable drivers. SunOS 4 IIRC. ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-29 21:37 [TUHS] shared objects in Unix Paul Winalski ` (2 preceding siblings ...) 2018-03-29 23:11 ` Dave Horsfall @ 2018-03-30 0:40 ` Clem Cole 2018-03-30 1:35 ` Charles H. Sauer ` (2 more replies) 3 siblings, 3 replies; 22+ messages in thread From: Clem Cole @ 2018-03-30 0:40 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 1461 bytes --] On Thu, Mar 29, 2018 at 5:37 PM, Paul Winalski <paul.winalski at gmail.com> wrote: > Also, when was symbol preemption invented? Traditional shared library > designs such as in IBM System/370, VMS, and Windows NT doesn't have > it. As one who worked on optimizations in compilers, I came to hate > symbol preemption because it prohibits many useful optimizations. ELF > does provide a way to turn it off, but it's on by default--you have to > explicitly declare symbols as protected or hidden via source language > pragmas to get rid of it. Unless it came from a place like Sun or Sun where Larry or Charlie might remember, I suspect that Steve Johnson is probably best to answer this part of your question -- assuming that it was created during his time in the compiler team in Summit. But, I don't remember when it came on to the scene frankly because it did not effect me. I think it might have been in the original COFF which came from those days, but its possible its from one of the many bastardization of COFF that occurred after its birth. I don't remember it being in any of the a.out flavors and I don't think macho has it. As an OS guy, all I remember about it frankly is you and some the compiler folks b*tching about it as a misfeature of UNIX at lunch ;-) ᐧ -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/caefc951/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 0:40 ` Clem Cole @ 2018-03-30 1:35 ` Charles H. Sauer 2018-03-30 2:10 ` Larry McVoy 2018-03-30 3:00 ` Ron Natalie 2018-03-30 21:53 ` Steve Johnson 2 siblings, 1 reply; 22+ messages in thread From: Charles H. Sauer @ 2018-03-30 1:35 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 2281 bytes --] > On Mar 29, 2018, at 7:40 PM, Clem Cole <clemc at ccc.com> wrote: > > > > On Thu, Mar 29, 2018 at 5:37 PM, Paul Winalski <paul.winalski at gmail.com <mailto:paul.winalski at gmail.com>> wrote: > Also, when was symbol preemption invented? Traditional shared library > designs such as in IBM System/370, VMS, and Windows NT doesn't have > it. As one who worked on optimizations in compilers, I came to hate > symbol preemption because it prohibits many useful optimizations. ELF > does provide a way to turn it off, but it's on by default--you have to > explicitly declare symbols as protected or hidden via source language > pragmas to get rid of it. > > Unless it came from a place like Sun or Sun where Larry or Charlie might remember, I suspect that Steve Johnson is probably best to answer this part of your question -- assuming that it was created during his time in the compiler team in Summit. > > But, I don't remember when it came on to the scene frankly because it did not effect me. I think it might have been in the original COFF which came from those days, but its possible its from one of the many bastardization of COFF that occurred after its birth. I don't remember it being in any of the a.out flavors and I don't think macho has it. > > As an OS guy, all I remember about it frankly is you and some the compiler folks b*tching about it as a misfeature of UNIX at lunch ;-) > > > ᐧ I’m not sure if Clem meant to type “Sun or IBM where Larry or Charlie” … I don’t readily find any documentation or history older than AIX 4.2, well beyond my tenure, but I believe we had shared libraries from the very beginning with AIX on the RT, presumably based on a.out. My recollection is that this was driven by (late) Larry Loucks, with assistance from Jack O’Quin and several of the ISC folks. Charlie -- voice: +1.512.784.7526 e-mail: sauer at technologists.com <mailto:sauer at technologists.com> fax: +1.512.346.5240 web: http://technologists.com/sauer/ <http://technologists.com/sauer/> Facebook/Google/Skype/Twitter: CharlesHSauer -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/4029b08e/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 1:35 ` Charles H. Sauer @ 2018-03-30 2:10 ` Larry McVoy 2018-03-30 2:34 ` Charles H. Sauer 0 siblings, 1 reply; 22+ messages in thread From: Larry McVoy @ 2018-03-30 2:10 UTC (permalink / raw) On Thu, Mar 29, 2018 at 08:35:50PM -0500, Charles H. Sauer wrote: > > > > On Mar 29, 2018, at 7:40 PM, Clem Cole <clemc at ccc.com> wrote: > > > > > > > > On Thu, Mar 29, 2018 at 5:37 PM, Paul Winalski <paul.winalski at gmail.com <mailto:paul.winalski at gmail.com>> wrote: > > Also, when was symbol preemption invented? Traditional shared library > > designs such as in IBM System/370, VMS, and Windows NT doesn't have > > it. As one who worked on optimizations in compilers, I came to hate > > symbol preemption because it prohibits many useful optimizations. ELF > > does provide a way to turn it off, but it's on by default--you have to > > explicitly declare symbols as protected or hidden via source language > > pragmas to get rid of it. > > > > ???Unless it came from a place like Sun or Sun where Larry or Charlie might remember, I suspect that Steve Johnson is probably best to answer this part of your question -- assuming that it was created during his time in the compiler team in Summit. > > > > But, I don't remember when it came on to the scene frankly because it did not effect me. I think it might have been in the original COFF which came from those days, but its possible its from one of the many bastardization of COFF that occurred after its birth. I don't remember it being in any of the a.out flavors and I don't think macho has it. > > > > As an OS guy, all I remember about it frankly is you and some the compiler folks b*tching about it as a misfeature of UNIX at lunch ;-) > > > > ??? > > ??? > > > I???m not sure if Clem meant to type ???Sun or IBM where Larry or Charlie??? ??? > > I don???t readily find any documentation or history older than AIX 4.2, well beyond my tenure, but I believe we had shared libraries from the very beginning with AIX on the RT, presumably based on a.out. My recollection is that this was driven by (late) Larry Loucks, with assistance from Jack O???Quin and several of the ISC folks. What was the underlying technology that enabled them in AIX? ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 2:10 ` Larry McVoy @ 2018-03-30 2:34 ` Charles H. Sauer 2018-03-30 3:04 ` Warner Losh 2018-03-30 20:33 ` Charles H Sauer 0 siblings, 2 replies; 22+ messages in thread From: Charles H. Sauer @ 2018-03-30 2:34 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 2913 bytes --] > On Mar 29, 2018, at 9:10 PM, Larry McVoy <lm at mcvoy.com> wrote: > > On Thu, Mar 29, 2018 at 08:35:50PM -0500, Charles H. Sauer wrote: >> >> >>> On Mar 29, 2018, at 7:40 PM, Clem Cole <clemc at ccc.com> wrote: >>> >>> >>> >>> On Thu, Mar 29, 2018 at 5:37 PM, Paul Winalski <paul.winalski at gmail.com <mailto:paul.winalski at gmail.com>> wrote: >>> Also, when was symbol preemption invented? Traditional shared library >>> designs such as in IBM System/370, VMS, and Windows NT doesn't have >>> it. As one who worked on optimizations in compilers, I came to hate >>> symbol preemption because it prohibits many useful optimizations. ELF >>> does provide a way to turn it off, but it's on by default--you have to >>> explicitly declare symbols as protected or hidden via source language >>> pragmas to get rid of it. >>> >>> ???Unless it came from a place like Sun or Sun where Larry or Charlie might remember, I suspect that Steve Johnson is probably best to answer this part of your question -- assuming that it was created during his time in the compiler team in Summit. >>> >>> But, I don't remember when it came on to the scene frankly because it did not effect me. I think it might have been in the original COFF which came from those days, but its possible its from one of the many bastardization of COFF that occurred after its birth. I don't remember it being in any of the a.out flavors and I don't think macho has it. >>> >>> As an OS guy, all I remember about it frankly is you and some the compiler folks b*tching about it as a misfeature of UNIX at lunch ;-) >>> >>> ??? >>> ??? >> >> >> I???m not sure if Clem meant to type ???Sun or IBM where Larry or Charlie??? ??? >> >> I don???t readily find any documentation or history older than AIX 4.2, well beyond my tenure, but I believe we had shared libraries from the very beginning with AIX on the RT, presumably based on a.out. My recollection is that this was driven by (late) Larry Loucks, with assistance from Jack O???Quin and several of the ISC folks. > > What was the underlying technology that enabled them in AIX? I didn’t pay much attention to this at the time, and don’t remember specifics. Given the change in focus from RT to RS/6000 in the transition from AIX 2 to AIX 3, and all of the other changes that occurred in AIX 3, I assume we started with something very primitive in AIX 1 and re-implemented for AIX 3. I’ve sent Jack a note about this discussion. Unless he or ISC folks chime in, or I find someone else to comment or provide documentation, I probably can’t add more. Rick Simpson wrote an article for Byte that might have something about this. He probably contributed to the initial design and (presumed) re-design. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/2e86423c/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 2:34 ` Charles H. Sauer @ 2018-03-30 3:04 ` Warner Losh 2018-03-30 20:33 ` Charles H Sauer 1 sibling, 0 replies; 22+ messages in thread From: Warner Losh @ 2018-03-30 3:04 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 3653 bytes --] On Thu, Mar 29, 2018 at 8:34 PM, Charles H. Sauer <sauer at technologists.com> wrote: > > > On Mar 29, 2018, at 9:10 PM, Larry McVoy <lm at mcvoy.com> wrote: > > On Thu, Mar 29, 2018 at 08:35:50PM -0500, Charles H. Sauer wrote: > > > > On Mar 29, 2018, at 7:40 PM, Clem Cole <clemc at ccc.com> wrote: > > > > On Thu, Mar 29, 2018 at 5:37 PM, Paul Winalski <paul.winalski at gmail.com < > mailto:paul.winalski at gmail.com <paul.winalski at gmail.com>>> wrote: > Also, when was symbol preemption invented? Traditional shared library > designs such as in IBM System/370, VMS, and Windows NT doesn't have > it. As one who worked on optimizations in compilers, I came to hate > symbol preemption because it prohibits many useful optimizations. ELF > does provide a way to turn it off, but it's on by default--you have to > explicitly declare symbols as protected or hidden via source language > pragmas to get rid of it. > > ???Unless it came from a place like Sun or Sun where Larry or Charlie > might remember, I suspect that Steve Johnson is probably best to answer > this part of your question -- assuming that it was created during his time > in the compiler team in Summit. > > But, I don't remember when it came on to the scene frankly because it did > not effect me. I think it might have been in the original COFF which came > from those days, but its possible its from one of the many bastardization > of COFF that occurred after its birth. I don't remember it being in any > of the a.out flavors and I don't think macho has it. > > As an OS guy, all I remember about it frankly is you and some the compiler > folks b*tching about it as a misfeature of UNIX at lunch ;-) > > ??? > ??? > > > > I???m not sure if Clem meant to type ???Sun or IBM where Larry or > Charlie??? ??? > > I don???t readily find any documentation or history older than AIX 4.2, > well beyond my tenure, but I believe we had shared libraries from the very > beginning with AIX on the RT, presumably based on a.out. My recollection is > that this was driven by (late) Larry Loucks, with assistance from Jack > O???Quin and several of the ISC folks. > > > What was the underlying technology that enabled them in AIX? > > > I didn’t pay much attention to this at the time, and don’t remember > specifics. > > Given the change in focus from RT to RS/6000 in the transition from AIX 2 > to AIX 3, and all of the other changes that occurred in AIX 3, I assume we > started with something very primitive in AIX 1 and re-implemented for AIX > 3. > Wikipedia says AIX v3 did shared libraries in 1990. SunOS 4.0 had them in 1988, also from wikipedia (though I can confirm 4.0 had shared libraries now). I recall vaguely working with them in 1991 when porting OI from SunOS to AIX 3.mumble, iirc. xlC was the weirdest C++ compiler at the time (which effectively means it wasn't cfront based). But I can't say with absolute certainty.... I’ve sent Jack a note about this discussion. Unless he or ISC folks chime > in, or I find someone else to comment or provide documentation, I probably > can’t add more. Rick Simpson wrote an article for Byte that might have > something about this. He probably contributed to the initial design and > (presumed) re-design. > I'd love to see something more definitive than my memory at the time + vague backup from Wikipedia... http://ps-2.kev009.com/rs6000/aix_ps_pdf/programming/using_shared_libraries.pdf is a primary source, but talks of AIX 4.1 Warner -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/cfd3fbdc/attachment-0001.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 2:34 ` Charles H. Sauer 2018-03-30 3:04 ` Warner Losh @ 2018-03-30 20:33 ` Charles H Sauer 1 sibling, 0 replies; 22+ messages in thread From: Charles H Sauer @ 2018-03-30 20:33 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 2305 bytes --] On Thu, Mar 29, 2018 at 09:35PM -0500, Charles H. Sauer wrote: On Mar 29, 2018, at 9:10 PM, Larry McVoy <lm at mcvoy.com> wrote: On Thu, Mar 29, 2018 at 08:35:50PM -0500, Charles H. Sauer wrote: I don’t readily find any documentation or history older than AIX 4.2, well beyond my tenure, but I believe we had shared libraries from the very beginning with AIX on the RT, presumably based on a.out. My recollection is that this was driven by (late) Larry Loucks, with assistance from Jack O’Quin and several of the ISC folks. What was the underlying technology that enabled them in AIX? I didn’t pay much attention to this at the time, and don’t remember specifics. Given the change in focus from RT to RS/6000 in the transition from AIX 2 to AIX 3, and all of the other changes that occurred in AIX 3, I assume we started with something very primitive in AIX 1 and re-implemented for AIX 3. I’ve sent Jack a note about this discussion. Unless he or ISC folks chime in, or I find someone else to comment or provide documentation, I probably can’t add more. Rick Simpson wrote an article for Byte that might have something about this. He probably contributed to the initial design and (presumed) re-design. Jack responded, credits Larry regarding AIX 1 & 2 and adds a little about AIX 3 re-implementation: Larry was definitely the driving force behind shared library support in AIX on the RT. While I did some minor compiler optimization work in that timeframe, I didn't work on the linker, which was the traditional Unix "ld" implementation, ported by ISC. I don't remember what, if any, special work they may have done to support shared libraries. I had more to do with the RS/6000 implementation. That linker had been written at Yorktown by Greg Chaiten (of information theory fame). It was highly optimized for fast loading with shared libraries. All load-time external references were collected into a Table of Contents that fit in a few contiguous pages. The rest of the executable was read-only, with pages that could be shared and reused by multiple processes. -- joq -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180330/0735731f/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 0:40 ` Clem Cole 2018-03-30 1:35 ` Charles H. Sauer @ 2018-03-30 3:00 ` Ron Natalie 2018-03-30 21:53 ` Steve Johnson 2 siblings, 0 replies; 22+ messages in thread From: Ron Natalie @ 2018-03-30 3:00 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 396 bytes --] I’m fairly sure Unix didn’t invent the dynamic linking symbol preemption. It was fundamental in the UNIVAC Exec8 operating system. EXEC8 also predates UNIX on the concept of fork() (or as they call it ER FORK$). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180329/9f44f4e4/attachment.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
* [TUHS] shared objects in Unix 2018-03-30 0:40 ` Clem Cole 2018-03-30 1:35 ` Charles H. Sauer 2018-03-30 3:00 ` Ron Natalie @ 2018-03-30 21:53 ` Steve Johnson 2 siblings, 0 replies; 22+ messages in thread From: Steve Johnson @ 2018-03-30 21:53 UTC (permalink / raw) [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #1: Type: text/plain, Size: 3412 bytes --] ----- Original Message ----- From: "Clem Cole" <clemc@ccc.com> To:"Paul Winalski" <paul.winalski at gmail.com> Cc:"The Eunuchs Hysterical Society" <tuhs at tuhs.org> Sent:Thu, 29 Mar 2018 20:40:36 -0400 Subject:Re: [TUHS] shared objects in Unix .... Unless it came from a place like Sun or Sun where Larry or Charlie might remember, I suspect that Steve Johnson is probably best to answer this part of your question -- assuming that it was created during his time in the compiler team in Summit. I think shared libraries were already in the code base (although perhaps not released) when I came to Summit in 1982. One of my main motivations was to make a product out of C++, which already looked like a winner in Research. One of the first things I did was to ship cfront -- this was a RATFOR like extension that translated C++ into C and then compiled it. Writing C++ with it wasn't too bad, because it mapped the C++ line numbers into the C file, but debugging it was ghastly. The mapping to C involved long names that had encoded information about the type and class membership and other gook, making the debugger almost worthless. (We were also shipping Pascal, Ada, and Fortran, which had lesser versions of the same problem). So we immediately set off to improve the debugging. I think shared libraries was based on an implementation done in a development area rather than Research, but it and COFF got a lot of tweaking to support Elf and Dwarf. A real C++ compiler, based on the PCC2 backend, came along as well, about the time we got the debugging figured out, and the final product was, IMHO, pretty respectable. A word about why I went to Summit. When I started managing, my interest in technology transfer ramped up quickly. As far as AT&T was concerned, the research culture and the development culture were very different. And the only technology transfer that seemed to work involved moving people. Reseach people had the best job in the world, and didn't want to move. But I found that, more and more, I was enjoying writing code that people liked to use much more than I enjoyed giving papers to 200 people who couldn't wait for me to sit down so they could talk about what they were doing. So I decided to give it a try. I still remember my first staff meeting with my department's supervisors. In 15 years in Reasearch, I never had heard anyone commit to deliver a given feature at a given time! And this happened many times in that meeting. By the end, I realized that I wasn't in Kansas any more... I learned tons from my supervisors, and I think they learned a lot from me. It was altogether a wonderful experience, except for the fact that AT&T continually demonstrated its complete ignorance of the software marketplace by hiring clueless people. At one point, a documentation person wrote up so documentation based on fodder we have given them. When it came back, it was completely incoherent. I gave it to my supervisors and they didn't get it either. Suddenly one said. "My God! They think the Fortran Optimizer is a piece of hardware!" Sure enough, from that point of view it made sense despite its total isolation from the real world. Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://minnie.tuhs.org/pipermail/tuhs/attachments/20180330/8e297379/attachment-0001.html> ^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2018-04-03 15:49 UTC | newest] Thread overview: 22+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-03-29 21:37 [TUHS] shared objects in Unix Paul Winalski 2018-03-29 22:01 ` Henry Bent 2018-03-29 22:26 ` Warner Losh 2018-03-29 23:11 ` Dave Horsfall 2018-03-29 23:22 ` Warner Losh 2018-03-29 23:24 ` Larry McVoy 2018-03-30 0:22 ` Clem Cole 2018-03-30 1:46 ` Larry McVoy 2018-03-30 4:28 ` Clem cole 2018-03-30 20:52 ` Jon Forrest 2018-03-30 22:42 ` Clem Cole 2018-03-30 23:29 ` Paul Winalski 2018-04-03 15:49 ` Derek Fawcus 2018-03-30 1:20 ` Lyndon Nerenberg 2018-03-30 0:40 ` Clem Cole 2018-03-30 1:35 ` Charles H. Sauer 2018-03-30 2:10 ` Larry McVoy 2018-03-30 2:34 ` Charles H. Sauer 2018-03-30 3:04 ` Warner Losh 2018-03-30 20:33 ` Charles H Sauer 2018-03-30 3:00 ` Ron Natalie 2018-03-30 21:53 ` Steve Johnson
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).