Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

[Noel Chiappa]

    > From: Jim Geist

    > When did mmap(2) come about?

Pretty sure it's a Berserkleyism. I think it came in with the VM stuff that
DARPA mandated for VAX Unix (for the research project they funded).

   Noel

Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

[Larry McVoy]

On Mon, Aug 17, 2020 at 03:27:15PM -0400, Noel Chiappa wrote:
>     > From: Jim Geist
> 
>     > When did mmap(2) come about?
> 
> Pretty sure it's a Berserkleyism. I think it came in with the VM stuff that
> DARPA mandated for VAX Unix (for the research project they funded).

Bill Joy imagined it, the prototype is in one the 4.x BSD releases.
Sun (Joe Moran) actually implemented it first in any Unix variant.
It's possible the concept existed in some other OS but I'm not aware
of it.

Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

[Dan Halbert]

On 8/17/20 3:30 PM, Larry McVoy wrote:
> On Mon, Aug 17, 2020 at 03:27:15PM -0400, Noel Chiappa wrote:
>>      > From: Jim Geist
>>
>>      > When did mmap(2) come about?
>>
>> Pretty sure it's a Berserkleyism. I think it came in with the VM stuff that
>> DARPA mandated for VAX Unix (for the research project they funded).
> Bill Joy imagined it, the prototype is in one the 4.x BSD releases.
> Sun (Joe Moran) actually implemented it first in any Unix variant.
> It's possible the concept existed in some other OS but I'm not aware
> of it.
I have a clear memory of having a discussion with Bill Joy about the 
idea of vread(2) and vwrite(2) when we were both grad students at 
Berkeley. I remember we were eating sausages from Top Dog and sitting 
outside Etcheverry Hall on or near the grassy plaza. I think vfork may 
have already existed, and we were talking about adding some kind of 
memory-mapped file I/O. I think I suggested the actual names, as a 
parallel to vfork. Some of my thinking might have come from my 
experience with TENEX while working summers at BBN. I was just a 
sounding board; I wasn't implementing any of this. vread and vwrite were 
in BSD4.1, but dropped in 4.2 (so my Googling says). I think it became 
clear later that mmap was an easier concept than the initial vread and 
vwrite ideas.

Dan H.

Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

[Paul Winalski]

On 8/17/20, Larry McVoy <lm@mcvoy.com> wrote:
> On Mon, Aug 17, 2020 at 03:27:15PM -0400, Noel Chiappa wrote:
[regarding mmap(2)]
>
> Bill Joy imagined it, the prototype is in one the 4.x BSD releases.
> Sun (Joe Moran) actually implemented it first in any Unix variant.
> It's possible the concept existed in some other OS but I'm not aware
> of it.
>
VAX/VMS had the equivalent of mmap(2) back in 1978.  You can specify a
range of contiguous pages in virtual memory and associate that with a
(page-aligned) range of blocks in a file.  The blocks in the file act
as backing store for the virtual memory.  VMS also has a system call
$CRETVA (create virtual address space) that lets you associate a VA
range using the system page file as backing store.  The VMS image
activator (runtime loader in Unix-speak) used these primitives to load
program images into virtual memory.  More than one process can map the
same region of a file.  This is how sharing of read-only program
segments such as .text is implemented.

I think Burroughs OSes had this concept even before VMS.

There is also $EXPREG (expand address region), which is more or less
equivalent to sbrk().

-Paul W.

Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

[Arthur Krewat]

On 8/17/2020 3:50 PM, Paul Winalski wrote:
> VAX/VMS had the equivalent of mmap(2) back in 1978.  You can specify a
> range of contiguous pages in virtual memory and associate that with a
> (page-aligned) range of blocks in a file.  The blocks in the file act
> as backing store for the virtual memory.
I, and my boss, took advantage of that while converting his database 
software from TOPS-10 to VMS.

His TOPS-10 implementation used various ways to optimize I/O from/to 
disk. On VMS, our benchmarks showed that as long as we understood the 
paging size, performance was about the same. RP06 vs. I think something 
bigger. Maybe RP07. But even still, it proved the operating system 
virtual memory system was up to the task.

I seem to remember there was a way to sync a page after you wrote it 
just to make sure it was committed to disk. It worked so well that I 
also seem to recall that we used some of that file space as heap.

art k.

Re: [TUHS] Memory management in Dennis Ritchie's C Compiler

[Rob Gingell]

On 8/17/2020 12:27 PM, Noel Chiappa wrote:
>      > From: Jim Geist
> 
>      > When did mmap(2) come about?
> 
> Pretty sure it's a Berserkleyism. I think it came in with the VM stuff that
> DARPA mandated for VAX Unix (for the research project they funded).

[What follows verges on being COFF-ish].

mmap() is descended from TENEX's PMAP, which is a simplified descendant 
of the file/process memory integration provided by Multics. (Other 
antecedents I leave to more informed genealogists than I.)

A Multics process had an address space made up of segments (files) which 
were paged (vs. swapped). Segments were what programs dealt with, paging 
was done to manage the physical resources of the system.

TENEX simplified everything to pages, address spaces were composed by 
mapping pages of the address space to files. Programs operated on pages, 
which were also what the system used to manage memory. Files were 
described by page tables, address spaces were described by page tables, 
lots of symmetry around things of size 512 (9 bits) (page size, page 
table size, page-tables of page tables for long files). File size and 
file system system capacities were announced in pages. Very harmonious, 
but uniquely so.

Multics was most definitely not portable due to its unique hardware 
constraints (Intel's 432 being an exception). TENEX only required paging 
support but was not portable from PDP-10s which were not keeping up with 
the world's movement to computers that were increasingly 4-bits shy of a 
word (i.e., not 36 bits). Many more considerations apply but to keep it 
only verging on COFF-ish will stop there -- going deeper would be best 
done in person with tankards of ale.

TENEX systems were ARPA's workhorse systems in the 1970s, supporting 
numerous research programs and sites. The file/process memory 
integration was a valued capability among the community (though, 
honestly, it was one of these things everyone said should exist though 
few actually used directly). Even though DEC provided commercial support 
by adopting TENEX as TOPS-20 and selling 1000s of them (vs. the 10s of 
them made via BBN + PARC's MAXC), as the 70's progressed it was apparent 
that things were going to shift elsewhere. DEC was hesitant about 
further 36-bit development and UNIX's arrival provided an intellectual 
migration path for the community that commercial alternatives didn't. If 
only UNIX could just accumulate some of the capabilities expected in the 
ARPA world, like file/process memory integration (ironically, one of the 
Multics complexities from which UNIX had departed.)

mmap() is a simplified, portable form of PMAP. It was written into the 
specifications for ARPA via Bill Joy as Larry McVoy and Noel Chiappa 
noted. It existed only as a specification for quite a while. First 
implementations were as a device-driver level function for things like 
frame-buffers for use by window systems and perhaps for other devices 
I'm not remembering at the moment.

The general file/process memory integration aspect didn't happen prior 
to SunOS 4.0 though I seem to recall some special-case subsets being 
tried by other UNIX implementations and also more generally with Mach, I 
believe. The gestalt of file/process memory integration in 4.0 was very 
TENEX-like in how it was used in fork()/exec()/brk()/shared libraries. 
4.0 differed from TENEX in being portable, launched on heterogeneous 
hardware (VAX, 68010 (Sun-2's), 68020 (Sun-3's), 386 (386i), SPARC 
(Sun-4) all except the VAX being actively supported Sun products at the 
time), and a general-networking view of how coherency should work (vs. a 
clustered-network such as TOPS-20's CFS.)

(As a hysterical note, every release of (BSD-based) SunOS through 4.0 
was always ported to the VAX as a portability check. It also provied us 
utility as a convenient way to quickly check for "but it works on the 
VAX" problems with new things gotten from UCB. Prior to 4.0, given that 
the BSD-based implementation basically tried to mimic a VAX internally, 
that wasn't too bad. But the VAXectomy that occurred in 4.0 made that 
much more daunting, and so the 4.0 VAX port was a pretty half-hearted 
attempt and the last new task for the 11/750 that faded deeper and 
deeper into the back of the lab over time.)

Re: [TUHS] mmap() (was: Memory management in Dennis Ritchie's C Compiler)

[Greg A. Woods]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

At Mon, 17 Aug 2020 17:52:08 -0700, Rob Gingell <gingell@computer.org> wrote:
Subject: Re: [TUHS] Memory management in Dennis Ritchie's C Compiler
>
> mmap() is descended from TENEX's PMAP, which is a simplified
> descendant of the file/process memory integration provided by
> Multics. (Other antecedents I leave to more informed genealogists than
> I.)

Indeed.  I see mmap() as just a poor-man's version of Multics segments,
and with it being bolted on after the fact, and without integrated
language and compiler support, that makes it far less useful.

> Multics was most definitely not portable due to its unique hardware
> constraints (Intel's 432 being an exception).

Well the basic design of Multics would/does fit reasonably well onto the
iAPX-386 / IA-32 architecture, though a lone x86 CPU would have been a
bit more limited in some capabilities than the Multics hardware was.

I think that if more than just OS/2 had made good use of all of the
segmentation features of iAPX-386 then possibly CPU designers (including
beyond Intel) would have had some incentive to continue to improve it
and make it more efficient and effective and indeed to carry it on in
full-fledged manner to the x86-64 architecture as well.

I think the failure of the iAPX-432 to meet performance expectations,
especially with silicon limitations of the day (along with the i960MX
never really getting full public exposure), put OS people in general off
the idea of building anything better than Unix, which of course only
needs a single flat address space and thus ran just fine on the much
better performing silicon already available at the same time.

I personally also think another problem that limited the porting and
evolution of Multics, and the creation of new systems using much more of
its design (and thus which favoured the growth of Unix in many ways) was
that a lot of commercial OS people were put off by the idea of going
with an object oriented model all the way from the languages on top and
down through the OS and into the hardware.  Too many users were (are)
still running applications written in FORTRAN and Cobol and similar, and
these were already implemented to use more direct read/write I/O models
for data access that Unix supports well.

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[-- Attachment #2: OpenPGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]