mmap

mmap

[Russ Cox]

Mmap means many things to many people.

Using mmap is most often not a matter of
performance as much as it is a matter of
flexibility: being able to mmap files is about
as close as most operating systems get to
exposing the underlying page table hardware,
which lets applications that aren't happy with
the OS-provided interface essentially design
their own.  The canonical example is databases,
which would really prefer to be operating systems
except for the whole having to write hardware
drivers thing.  9vx is another example.

If all you want is to map a few read-only (or at least
non-write-back) files, like you'd need to implement
dynamically-loaded shared libraries, that's not too hard
to add.  Back when I did the first cut of linuxemu,
I changed segattach to accept 'file!a.out' as a segment name.
If you've got a lot of these, you might need to
change the segment array to be larger or perhaps
dynamically sized.  The particular interface I chose
felt clumsy so I never even suggested putting it into
the kernel, but mostly you can reuse the code that
demand-loads ordinary Plan 9 binaries.

The shared-library kind of mmap is by far the simplest
mmap use case.  Even allowing mmap as a surrogate
for malloc would probably push you pretty far toward
needing to rewrite the memory system, since the
mmap-based mallocs tend to use a large number of
small mappings.

True mmap support in the Plan 9 kernel would
require rewriting the entire memory system to
work in terms of pages instead of the coarser
regions the kernel calls segments (text, data, bss,
stack).  This would be a lot of work with almost
zero benefit.  I'd rather let those applications not
happy with what Plan 9 provides just not use Plan 9.
There's no need to be all things to all people.

Also, on Plan 9, shared memory means that the
memory at the time of rfork is shared.  If one proc
then does a segattach, the other proc sharing its
memory will not see the segattach.  The same is
true of segdetach.  This is another reason that you
can't just reuse the existing mechanisms to hack
in a traditional full-featured mmap.  To really get
those right you might need (gasp) TLB shootdowns,
which would be even more work.

It's nice that Plan 9's memory system is simple--just
a few segments, each of which is a run of pages.
"True" mmap support would destroy that.  Please don't.

Russ

Re: [9fans] mmap

[erik quanstrom]

> in a traditional full-featured mmap

i've noticed that some combinations of words
are scarier than others.

☺

- erik

Re: [9fans] mmap

[Enrico Weigelt]

* Russ Cox <rsc@swtch.com> wrote:

Hi,

<big_snip />

I don't know much of native plan9 (only using plan9port), but IMHO
an full mmap() is a really nice thing. It can make a lot things
easier if you just map the whole file into the process' memory and
let the kernel handle the actual IO.

Some time ago I hacked up a little XML-based database (just to
show myself that it needn't to be that ugly slow as Xindice ;-)),
and being able to conveniently work with pointers instead of ugly
seek() made really fun ;-P


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Charles Forsyth]

> It can make a lot things
> easier if you just map the whole file into the process' memory and
> let the kernel handle the actual IO.

the word "superficially" should be in there somewhere.

Re: [9fans] mmap

[Alexander Sychev]

On Tue, 29 Jul 2008 12:52:14 +0400, Enrico Weigelt <weigelt@metux.de>
wrote:

Hi!

> an full mmap() is a really nice thing. It can make a lot things
> easier if you just map the whole file into the process' memory and
> let the kernel handle the actual IO.

Yes, it is comfortable. But just think a bit - what will you do in the
mmap implementation when you had mapped a remote file (in Plan9 you can't
be sure some file is local or it is really just a file), and the
connection has just been broken? Surprise!

For example, in Windows you will have an "access violation". It is very
funny to rewrite a code for using the good old file i/o instead of mmap.
In Linux you can't map a non-regular file.

Yes, you can read entire the file into the memory, but a size of the file
can be very big and you will destroy the main advantage of mmap - the
reading of only needed pieces of the file, not entire.

I beleive, mmap is not for distributed systems.

--
Best regards,
	santucco

Re: [9fans] mmap

[erik quanstrom]

> Yes, it is comfortable.

where's jim when you need him?

> But just think a bit - what will you do in the
> mmap implementation when you had mapped a remote file (in Plan9 you can't
> be sure some file is local or it is really just a file), and the
> connection has just been broken? Surprise!

you can't make the assumption that a file is local in *ix, either.

- erik

Re: [9fans] mmap

[ron minnich]

On Tue, Jul 29, 2008 at 8:19 AM, erik quanstrom <quanstro@coraid.com> wrote:

> you can't make the assumption that a file is local in *ix, either.
>

in fact, for the last 20 years, every program run on a sunos/solaris
machine has used mmap for the exec.

ron

Re: [9fans] mmap

[Roman V. Shaposhnik]

ron minnich wrote:
> On Tue, Jul 29, 2008 at 8:19 AM, erik quanstrom <quanstro@coraid.com> wrote:
>
>
>> you can't make the assumption that a file is local in *ix, either.
>>
>>
>
> in fact, for the last 20 years, every program run on a sunos/solaris
> machine has used mmap for the exec.
>
mmap() is everywhere on Solaris. Even when you do something as trivial
as sort(1) --
it'll try to mmap() the operands and only if that fails it is going to
use read()/write().

Thanks,
Roman.

P.S. Which is not that bad of an optimization if you ask me...

Re: [9fans] mmap

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Tue, Jul 29, 2008 at 8:04 AM, Alexander Sychev <santucco@gmail.com>wrote:

> On Tue, 29 Jul 2008 12:52:14 +0400, Enrico Weigelt <weigelt@metux.de>
> wrote:
>
> Hi!
>
>  an full mmap() is a really nice thing. It can make a lot things
>> easier if you just map the whole file into the process' memory and
>> let the kernel handle the actual IO.
>>
>
> Yes, it is comfortable. But just think a bit - what will you do in the mmap
> implementation when you had mapped a remote file (in Plan9 you can't be sure
> some file is local or it is really just a file), and the connection has just
> been broken? Surprise!


>
> For example, in Windows you will have an "access violation". It is very
> funny to rewrite a code for using the good old file i/o instead of mmap.
> In Linux you can't map a non-regular file.
>
> Yes, you can read entire the file into the memory, but a size of the file
> can be very big and you will destroy the main advantage of mmap - the
> reading of only needed pieces of the file, not entire.
>

And if you read it in in pieces, how can you be sure the file isn't changing
on the system it's hosted on while you're looking at it?


>
> I beleive, mmap is not for distributed systems.
>
> --
> Best regards,
>        santucco
>
>

[-- Attachment #2: Type: text/html, Size: 2128 bytes --]

Re: [9fans] mmap

[Venkatesh Srinivas]

As far as interfaces go, mmap() is pretty tragic - the underlying
translation structures can express more interesting things, some of
which are even worth doing.

There have even been OSes that let userland apps play with their address
spaces in far more interesting ways - KeyKOS and EROS come to mind. And
they were even fast, or something.

In a system like Plan 9, where your file servers are on the other side
of a 9P link, this mmap thing seems dubious. If what you want is the
convenience that you get from having all the bytes in memory, reading
them all in wouldn't be too hard. mmap()s magic really arises when you
have a page-cache-like-thing.

--vs

Re: [9fans] mmap

[Charles Forsyth]

> As far as interfaces go, mmap() is pretty tragic - the underlying
> translation structures can express more interesting things, some of
> which are even worth doing.

> There have even been OSes that let userland apps play with their address
> spaces in far more interesting ways

i think that's right, and that's the interesting case to investigate

Re: [9fans] mmap

[Charles Forsyth]

> i think that's right, and that's the interesting case to investigate

provided, of course, that you're interested in the applications that might use it.
otherwise it will just complicate things to no good effect.

Re: [9fans] mmap

[Roman V. Shaposhnik]

On Tue, 2008-07-29 at 12:28 -0400, Venkatesh Srinivas wrote:
> As far as interfaces go, mmap() is pretty tragic - the underlying
> translation structures can express more interesting things, some of
> which are even worth doing.

I can't agree more. The way I look at it is that mmap() seems to
be the answer but nobody ever bothered to ask the question it is
supposed to answer.

> In a system like Plan 9, where your file servers are on the other side
> of a 9P link, this mmap thing seems dubious. If what you want is the
> convenience that you get from having all the bytes in memory, reading
> them all in wouldn't be too hard. mmap()s magic really arises when you
> have a page-cache-like-thing.

As Russ, quite rightfully, pointed out: mmap() means different things
to different people. The tragic part is, that it tries to do lots of
things but it doesn't do anything particularly well. Personally, my
experience of trying to use mmap() as a useful abstraction for the
CPU's MMU was the last straw. It can't do even that reliably
and in a portable fashion. Not to digress, but I was even more surprised
to learn that there's not a single API on UNIX that can:
    http://thread.gmane.org/gmane.os.netbsd.devel.kernel/6392/focus=6457

So, what mmap() (the way it is done on UNIX) is good for? Here's my
personal list. Feel free to add (and suggest alternatives on systems
lacking mmap() such as Plan9):
   * a *lazy* way of handling highly irregular I/O over large files.
     Cases, where you can't really tell which parts of the file are
     going to be needed. The best example here is mmap() on exec.
     You don't have to read() all of .text if the actual execution path
     only takes you to a couple of routines.
   * an optimization for regular I/O. To some extent, I've always
     wondered why read always takes its second argument -- a lot of
     times I don't really care where the buffer with the data I need
     ends up in my address space.

That's pretty much it. Everything else, feels like a hack in a dire
need of a better abstraction.

Thanks,
Roman.

Re: [9fans] mmap

[Paul Lalonde]

Roman V. Shaposhnik wrote:
> Personally, my
> experience of trying to use mmap() as a useful abstraction for the
> CPU's MMU was the last straw. It can't do even that reliably
> and in a portable fashion. Not to digress, but I was even more surprised
> to learn that there's not a single API on UNIX that can:
>     http://thread.gmane.org/gmane.os.netbsd.devel.kernel/6392/focus=6457
>

Amen.  I've had to get our OS team to shoehorn in some remap() calls
that let me point two(many) virtual addresses to a common physical
address.  It's disgusting that the process for doing this with mmap
requires /proc/#/mem and other scary things.

Does anyone have pointers to *nice* interfaces for doing this?  I don't
care which OS, but I'd like to not have the implementation I'm
requesting suck because I missed some gotchas that someone else already hit.

Thanks,
    Paul

Re: [9fans] mmap

[Charles Forsyth]

> Does anyone have pointers to *nice* interfaces for doing this?  I don't
> care which OS, but I'd like to not have the implementation I'm
> requesting suck because I missed some gotchas that someone else already hit.

paul lalonde's request is a good one, but i think a useful qualification
is that it should also be shown in practice to have a reasonable implementation
on several agreeably different architectures.  one of the reasons mmap is
bad is that like job control it whacked some vague ideas into an existing
system without reference to its existing conventions, then vaguely extended
it in a vague way from time to time, vaguely different from platform to platform,
and with vague documentation.  ``could they be any more vague?''

Re: [9fans] mmap

[Bakul Shah]

On Thu, 31 Jul 2008 22:39:43 BST Charles Forsyth <forsyth@terzarima.net>  wrote:
> > Does anyone have pointers to *nice* interfaces for doing this?  I don't
> > care which OS, but I'd like to not have the implementation I'm
> > requesting suck because I missed some gotchas that someone else already hit.
>
> paul lalonde's request is a good one, but i think a useful qualification is
> that it should also be shown in practice to have a reasonable implementation
> on several agreeably different architectures.  one of the reasons mmap is
> bad is that like job control it whacked some vague ideas into an existing
> system without reference to its existing conventions, then vaguely extended
> it in a vague way from time to time, vaguely different from platform to
> platform, and with vague documentation.  ``could they be any more vague?''

I vaguely remember seeing a comment on Karels and McKusick's
original mmap proposal (about 20 years ago).  I think it was
by dmr.  Something to the effect that if such a facility is
desired it should be made as integral to the OS as an inode
is to Unix (and not bolted on the side). Makes for an
interesting thought experiment as potentially all sorts of
simplifications are possible.

Re: [9fans] mmap

[ron minnich]

here is a thought:

the kernel does mmap for code/data. This is because we think of a file
as a segment of data that somehow maps well to a segment of memory.

You wouldn't execute code from a stream, now, would you?

Well, this: http://www.ambric.com/

has hardware channels. And you can
call from channel
and execute code being sent down a channel to you from another cpu.

There's no real analogue to this in any OS I've used for a while ...

You could write the mcilroy sieve in this very directly. You could
even, when starting a new thread, push the code down to the next cpu.
And that cpu is paused in a call from channel until it gets its code.
A nice way to keep them idle until you need them.

it's a very interesting architecture, to say the least. For me anyway
the most novel thing I've seen in a while.

ron

Re: [9fans] mmap

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Thu, Jul 31, 2008 at 5:32 PM, ron minnich <rminnich@gmail.com> wrote:

> here is a thought:
>
> the kernel does mmap for code/data. This is because we think of a file
> as a segment of data that somehow maps well to a segment of memory.
>
> You wouldn't execute code from a stream, now, would you?
>
> Well, this: http://www.ambric.com/
>
> has hardware channels. And you can
> call from channel
> and execute code being sent down a channel to you from another cpu.


>
> There's no real analogue to this in any OS I've used for a while ...
>
> You could write the mcilroy sieve in this very directly. You could
> even, when starting a new thread, push the code down to the next cpu.
> And that cpu is paused in a call from channel until it gets its code.
> A nice way to keep them idle until you need them.
>
> it's a very interesting architecture, to say the least. For me anyway
> the most novel thing I've seen in a while.
>

The ARM 7 and ARM 9 procs in the Nintendo DS can talk back and forth via a
FIFO at a pretty low level :-)

http://www.double.co.nz/nintendo_ds/nds_develop7.html

Dave

>
> ron
>
>

[-- Attachment #2: Type: text/html, Size: 1931 bytes --]

Re: [9fans] mmap

[Richard Miller]

> has hardware channels. And you can
> call from channel
> and execute code being sent down a channel to you from another cpu.
...
> it's a very interesting architecture, to say the least. For me anyway
> the most novel thing I've seen in a while.

Interesting but not novel.  Remember the transputer, circa 1985?  Hardware
channels, both inter- and intra-chip.  The normal way of loading code was
to read it down a link from another processor - used for bootstrapping
an array of transputers, but also could be used for dynamic propagation
of processes through the network.

Re: [9fans] mmap

[ron minnich]

On Sat, Aug 2, 2008 at 6:22 AM, Richard Miller <9fans@hamnavoe.com> wrote:
>> has hardware channels. And you can
>> call from channel
>> and execute code being sent down a channel to you from another cpu.
> ...
>> it's a very interesting architecture, to say the least. For me anyway
>> the most novel thing I've seen in a while.
>
> Interesting but not novel.  Remember the transputer, circa 1985?  Hardware
> channels, both inter- and intra-chip.  The normal way of loading code was
> to read it down a link from another processor - used for bootstrapping
> an array of transputers, but also could be used for dynamic propagation
> of processes through the network.
>

yes, but IIRC the transputer did not have the ability to execute code
from a channel in the way the ambric does -- literally having a PC
that includes a channel ID. There are a few other wrinkles to the
ambric that make it a little neater than the transputer. Ambric is
pretty well aware of what the transputer did.

rno

Re: [9fans] mmap

[Richard Miller]

> Ambric is
> pretty well aware of what the transputer did.

Glad to hear it - there were some good ideas behind the transputer which
are worth recycling.  From your description it sounds like the ambric has
some interesting new refinements as well.

Re: [9fans] mmap

[Paweł Lasek]

On Wed, Jul 30, 2008 at 18:36, Roman V. Shaposhnik <rvs@sun.com> wrote:
> As Russ, quite rightfully, pointed out: mmap() means different things
> to different people. The tragic part is, that it tries to do lots of
> things but it doesn't do anything particularly well. Personally, my
> experience of trying to use mmap() as a useful abstraction for the
> CPU's MMU was the last straw. It can't do even that reliably
> and in a portable fashion. Not to digress, but I was even more surprised
> to learn that there's not a single API on UNIX that can:
>    http://thread.gmane.org/gmane.os.netbsd.devel.kernel/6392/focus=6457
>
> So, what mmap() (the way it is done on UNIX) is good for? Here's my
> personal list. Feel free to add (and suggest alternatives on systems
> lacking mmap() such as Plan9):
>   * a *lazy* way of handling highly irregular I/O over large files.
>     Cases, where you can't really tell which parts of the file are
>     going to be needed. The best example here is mmap() on exec.
>     You don't have to read() all of .text if the actual execution path
>     only takes you to a couple of routines.
>   * an optimization for regular I/O. To some extent, I've always
>     wondered why read always takes its second argument -- a lot of
>     times I don't really care where the buffer with the data I need
>     ends up in my address space.
>
> That's pretty much it. Everything else, feels like a hack in a dire
> need of a better abstraction.

SBCL uses it to kick out system vm out of managing it's memory. At
least the version I got
on linux calls mmap() with MAP_ANONYMOUS, rwx permissions and I think MAP_FIXED.

Unfortunately, when you disable overcommit it fails to start on my
machine, because I clearly don't have
8GB of free memory&swap for it (I total at 2.5 GB w/ swap). I wish
someone gave a plain interface to just
took control of memory space...

I also had seen a simple malloc() implementation (targeted at people
who want to write their own systems) that
had a simple form of mmap/unmap calls (without mapping of files) as
it's only requirement. If it's the only kernel-exported API for
managing memory, it can make sense. I think that was done in L4 API,
which organizes it's processes by address spaces (the distinction
between what we call a process and thread being only their address
spaces - a process was a set of threads grouped by address space). A
process could also implement it's own pager and create processes that
used it's address space (basically what 9vx does, as you could also
install appropriate syscall trampolines).

As a counter-example, MS Windows' Notepad.exe is one of the worst
offenders when it comes to mmap().
To put it simply, the reason it doesn't work on large files is that it
mmaps them. ALWAYS. It's even stated in IFS SDK,
that "implementation of memory mapping mechanism is required to get
notepad to work on the filesystem in question".

> Thanks,
> Roman.
>

-- 
Paweł Lasek

Re: [9fans] mmap

[Enrico Weigelt]

* Venkatesh Srinivas <me@acm.jhu.edu> wrote:

... redirecting back to 9fans ;-P

> As far as interfaces go, mmap() is pretty tragic - the underlying
> translation structures can express more interesting things, some of
> which are even worth doing.

Well, the biggest problem, IMHO are the incompatible semantics
between platforms, and some things IMHO aren't needed at all.

What I expect from an clean mmap() is:

* map in a given region of some file into process's address space
* mapping types:
    a) readonly
    b) readwrite
    c) private / copy-on-write
* either let the app or the kernel decide on start address
* optional hints on intended access patterns (eg. if it might
  be wise to do read-ahead)

Ah, and there should be a special (size-limited) kernel device
which holds evrything in RAM exclusively (never get swapped out)
for holding critical security information.

> There have even been OSes that let userland apps play with their address
> spaces in far more interesting ways - KeyKOS and EROS come to mind. And
> they were even fast, or something.

hmm, never heared about them. what are they doing at this point ?
>
> In a system like Plan 9, where your file servers are on the other
> side of a 9P link, this mmap thing seems dubious. If what you want is the
> convenience that you get from having all the bytes in memory, reading
> them all in wouldn't be too hard. mmap()s magic really arises when you
> have a page-cache-like-thing.

Well, mmap() is a tricky thing since it breaks common filesystem
semantics. So it can only be done, and only makes sense on specific
kind of files/devices which are only acessed block-wise. So the
absolute MUST is that the underlying file supports (at least block-
wise) random access - mmap()'ed streams are completely nonsense.

Convenience is one point (sometimes be a big point), but another
important one is sharing. Without mmap(), an (real) shared library
support most likely will require special kernel support.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Roman V. Shaposhnik]

On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
> Convenience is one point (sometimes be a big point), but another
> important one is sharing. Without mmap(), an (real) shared library
> support most likely will require special kernel support.

What aspect of shared libraries are you aching for? Dynamic
linking or the dynamic loading?

Thanks,
Roman.

Re: [9fans] mmap

[Enrico Weigelt]

* Roman V. Shaposhnik <rvs@sun.com> wrote:
> On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
> > Convenience is one point (sometimes be a big point), but another
> > important one is sharing. Without mmap(), an (real) shared library
> > support most likely will require special kernel support.
>
> What aspect of shared libraries are you aching for? Dynamic
> linking or the dynamic loading?

3rd: Sharing pages.

Well, this perhaps also could be done if the kernel would be able
to detect equal pages and automatically map them together (maybe
w/ copy-on-write again).


BTW mmap() is also nice for creating shared memory between
(local) processes. For example RDBMS'es can get a huge benetit
from this.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Joel C. Salomon]

On Wed, Jul 30, 2008 at 11:29 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> Convenience is one point (sometimes be a big point), but another
> important one is sharing. Without mmap(), an (real) shared library
> support most likely will require special kernel support.

Actually, almost any kernel support for shared libraries will need
something like mmap() internally.

I forget who said it, and the local firewall won't allow me to search
the online copy of /sys/games/lib/fortunes, but there should be a line
there about Linux having 200+ system calls, most of them emulatable
with mmap().

--Joel

Re: [9fans] mmap

[Kernel Panic]

Joel C. Salomon wrote:
> On Wed, Jul 30, 2008 at 11:29 AM, Enrico Weigelt <weigelt@metux.de> wrote:
>
>> Convenience is one point (sometimes be a big point), but another
>> important one is sharing. Without mmap(), an (real) shared library
>> support most likely will require special kernel support.
>>
>
> Actually, almost any kernel support for shared libraries will need
> something like mmap() internally.
>
> I forget who said it, and the local firewall won't allow me to search
> the online copy of /sys/games/lib/fortunes, but there should be a line
> there about Linux having 200+ system calls, most of them emulatable
> with mmap().
>
what nonsense is that?
> --Joel
>

cinap

Re: [9fans] mmap

[erik quanstrom]

> > I forget who said it, and the local firewall won't allow me to search
> > the online copy of /sys/games/lib/fortunes, but there should be a line
> > there about Linux having 200+ system calls, most of them emulatable
> > with mmap().
> >
> what nonsense is that?

/sys/games/lib/fortunes:4184: A handful of characteristics of Unix are responsible for its resilience. First, Unix is simple: whereas some operating systems implement thousands of system calls and have unclear design goals, Unix systems typically implement only hundreds of system calls and have a very clear design. -- Linux Kernel Development, 2nd Ed. by Robert Love

- erik

Re: [9fans] mmap

[Joel C. Salomon]

On Wed, Jul 30, 2008 at 12:25 PM, Joel C. Salomon
<joelcsalomon@gmail.com> wrote:
> I forget who said it,

Found it in <http://9fans.net/archive/2002/08/130>:

On Tue, 13 Aug 2002 07:43:45 -0400, David Gordon Hogan <dhog@pla...> wrote:
> > On freebsd and Linux, exec happens via an mmap (more or less).
<snip>
> So [gigantic leap here], not only does Linux have ~250 system
> calls, but most of them can be emulated with mmap?

--Joel