Re: [9fans] mmap

Re: [9fans] mmap

[erik quanstrom]

> * Roman V. Shaposhnik <rvs@sun.com> wrote:
> > On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
> > > Convenience is one point (sometimes be a big point), but another
> > > important one is sharing. Without mmap(), an (real) shared library
> > > support most likely will require special kernel support.
> >
> > What aspect of shared libraries are you aching for? Dynamic
> > linking or the dynamic loading?
>
> 3rd: Sharing pages.

segment(3) already provides this.

- erik

Re: [9fans] mmap

[ron minnich]

On Sun, Nov 2, 2008 at 5:50 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> * Roman V. Shaposhnik <rvs@sun.com> wrote:
>> > On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
>> > > Convenience is one point (sometimes be a big point), but another
>> > > important one is sharing. Without mmap(), an (real) shared library
>> > > support most likely will require special kernel support.
>> >
>> > What aspect of shared libraries are you aching for? Dynamic
>> > linking or the dynamic loading?
>>
>> 3rd: Sharing pages.
>
> segment(3) already provides this.


I still remember this science fiction movie from when I was a kid.
Time travelers. At the end of the movie, you realized that they were
ending at the point where they were started, stuck in a loop, oh no!

and here we are at mmap again.

ron

Re: [9fans] mmap

[Charles Forsyth]

>I still remember this science fiction movie from when I was a kid.

which one was that? it sounds more interesting than mmap.

Re: [9fans] mmap

[ron minnich]

On Sun, Nov 2, 2008 at 11:29 PM, Charles Forsyth <forsyth@terzarima.net> wrote:
>>I still remember this science fiction movie from when I was a kid.
>
> which one was that? it sounds more interesting than mmap.
>
>

I wish I could remember. It had the usual guys in silvery suits. They
walk through a frame and are back in time. Key point was, at the end,
that they ended up escaping but for reasons unknown, walking back
through the frame -- bad idea.

ron

Re: [9fans] mmap

[Bruce Ellis]

Is it worth adding that in Inferno all libraries are shared as is all
of memory - seamlessly and safely? Always has been - always will be.

And there is no bloody mmap() in sight.

Also I believe that every bad science fiction movie is analogous to a
computer science moral play. And the more the movie tries to make
sense the bigger the lesson to be learnt. Of course Plan9 chose the
worst movie for the best lessons-that-have-been-learnt.

Just making it up - I seem to be the last foreign 9fan in town.

brucee

On Mon, Nov 3, 2008 at 8:58 AM, ron minnich <rminnich@gmail.com> wrote:
> On Sun, Nov 2, 2008 at 11:29 PM, Charles Forsyth <forsyth@terzarima.net> wrote:
>>>I still remember this science fiction movie from when I was a kid.
>>
>> which one was that? it sounds more interesting than mmap.
>>
>>
>
> I wish I could remember. It had the usual guys in silvery suits. They
> walk through a frame and are back in time. Key point was, at the end,
> that they ended up escaping but for reasons unknown, walking back
> through the frame -- bad idea.
>
> ron
>
>

Re: [9fans] mmap

[Enrico Weigelt]

* ron minnich <rminnich@gmail.com> wrote:

> I wish I could remember. It had the usual guys in silvery suits. They
> walk through a frame and are back in time. Key point was, at the end,
> that they ended up escaping but for reasons unknown, walking back
> through the frame -- bad idea.

Time bandits ?


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Andrew Simmons]

I don't want to imply that Ron is quite such an old fart as me, but
somehow I don't get the impression that he was a kid in 1981, when
"Time Bandits" came out. Ron, if you could give some clue as to when
you saw the movie, I'm pretty sure that the group could mount a
co-ordinated effort to identify it.

2008/11/4 Enrico Weigelt <weigelt@metux.de>:
> * ron minnich <rminnich@gmail.com> wrote:
>
>> I wish I could remember. It had the usual guys in silvery suits. They
>> walk through a frame and are back in time. Key point was, at the end,
>> that they ended up escaping but for reasons unknown, walking back
>> through the frame -- bad idea.
>
> Time bandits ?
>
>
> cu
> --
> ----------------------------------------------------------------------
>  Enrico Weigelt, metux IT service -- http://www.metux.de/
>
>  cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>
>

Re: [9fans] mmap

[Bakul Shah]

On Tue, 04 Nov 2008 20:15:04 +1300 "Andrew Simmons" <kodogo@gmail.com>  wrote:
> I don't want to imply that Ron is quite such an old fart as me, but
> somehow I don't get the impression that he was a kid in 1981, when
> "Time Bandits" came out. Ron, if you could give some clue as to when
> you saw the movie, I'm pretty sure that the group could mount a
> co-ordinated effort to identify it.

Sounds like The Time Travelers

Re: [9fans] mmap

[Charles Forsyth]

"time travel" plots reminds me of an obscure but splendid czech film that i've only seen once
http://filmjournal.net/czech/2006/09/18/tomorrow-ill-wake-up-and-scald-myself-with-tea/
but have yet to find on DVD. it is very funny.

Re: [9fans] mmap

[ron minnich]

On Mon, Nov 3, 2008 at 11:15 PM, Andrew Simmons <kodogo@gmail.com> wrote:
> I don't want to imply that Ron is quite such an old fart as me, but
> somehow I don't get the impression that he was a kid in 1981, when
> "Time Bandits" came out. Ron, if you could give some clue as to when
> you saw the movie, I'm pretty sure that the group could mount a
> co-ordinated effort to identify it.
>

It had to be 1963.

I am doubtless older than you, my common experience nowadays is being
older than everyone.

But so much more charming!


ron

Re: [9fans] mmap

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 897 bytes --]

On Mon, Nov 3, 2008 at 8:43 PM, Enrico Weigelt <weigelt@metux.de> wrote:

> * ron minnich <rminnich@gmail.com> wrote:
>
> > I wish I could remember. It had the usual guys in silvery suits. They
> > walk through a frame and are back in time. Key point was, at the end,
> > that they ended up escaping but for reasons unknown, walking back
> > through the frame -- bad idea.
>
> Time bandits ?
>

Don't touch it... it's pure evil.


>
>
> cu
> --
> ----------------------------------------------------------------------
>  Enrico Weigelt, metux IT service -- http://www.metux.de/
>
>  cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>
>

[-- Attachment #2: Type: text/html, Size: 1524 bytes --]

Re: [9fans] mmap

[Kim Shrier]

On Nov 4, 2008, at 9:02 AM, ron minnich wrote:

> On Mon, Nov 3, 2008 at 11:15 PM, Andrew Simmons <kodogo@gmail.com>
> wrote:
>> I don't want to imply that Ron is quite such an old fart as me, but
>> somehow I don't get the impression that he was a kid in 1981, when
>> "Time Bandits" came out. Ron, if you could give some clue as to when
>> you saw the movie, I'm pretty sure that the group could mount a
>> co-ordinated effort to identify it.
>>
>
> It had to be 1963.
>
> I am doubtless older than you, my common experience nowadays is being
> older than everyone.
>
> But so much more charming!
>

I think the movie is The Time Travelers.

I remember seeing it in the early 60's.

Maybe I'm getting too old... Hey you kids! Get off of my lawn.

Kim

[9fans] Time travel

[Lyndon Nerenberg]

On Tue, 4 Nov 2008, Kim Shrier wrote:

> Maybe I'm getting too old... Hey you kids! Get off of my lawn.

Make that "Get off of my Wifi!"

--lyndon (in firm denial of becoming a legitimate 'grey beard')

Re: [9fans] Time travel

[Jack Johnson]

On Tue, Nov 4, 2008 at 3:09 PM, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
> Make that "Get off of my Wifi!"

Those crazy kids with their Hulu loops.

-J

Re: [9fans] Time travel

[Bruce Ellis]

No respect those kids wearing crazy clothes and mucking up constantly.

BTW can whoever took the photos of me and tiger please send them to me
or post them somewhere.

brucee

On Wed, Nov 5, 2008 at 2:38 AM, Jack Johnson <knapjack@gmail.com> wrote:
> On Tue, Nov 4, 2008 at 3:09 PM, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
>> Make that "Get off of my Wifi!"
>
> Those crazy kids with their Hulu loops.
>
> -J
>
>

Re: [9fans] mmap

[gas]

FYI:
http://thepiratebay.org/torrent/3835498/The_funniest_and_cleverest_time_travel_movie_ever_made__

/G.

--- Den tis 2008-11-04 skrev Charles Forsyth <forsyth@terzarima.net>:

> Från: Charles Forsyth <forsyth@terzarima.net>
> Ämne: Re: [9fans] mmap
> Till: 9fans@9fans.net
> Datum: tisdag 4 november 2008 15.13
> "time travel" plots reminds me of an obscure but
> splendid czech film that i've only seen once
> http://filmjournal.net/czech/2006/09/18/tomorrow-ill-wake-up-and-scald-myself-with-tea/
> but have yet to find on DVD. it is very funny.


      __________________________________________________________
Låna pengar utan säkerhet. Jämför vilkor online hos Kelkoo.
http://www.kelkoo.se/c-100390123-lan-utan-sakerhet.html?partnerId=96915014

Re: [9fans] mmap

[Charles Forsyth]

>> But now I'm curious how executables and shared libraries are
>> actually handled on plan9.
>>

>what's a shared library?

even if plan 9 were to have them, and i were doing it, i don't think i'd
implement them using mmap and trampolines.

Re: [9fans] mmap

[ron minnich]

On Sun, Nov 2, 2008 at 8:18 PM, Enrico Weigelt <weigelt@metux.de> wrote:

> But now I'm curious how executables and shared libraries are
> actually handled on plan9.
>

what's a shared library?

Executables:

/sys/src/9/

Check it out, it's short and sweet.

ron

Re: [9fans] mmap

[Enrico Weigelt]

* erik quanstrom <quanstro@quanstro.net> wrote:
> > * Roman V. Shaposhnik <rvs@sun.com> wrote:
> > > On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
> > > > Convenience is one point (sometimes be a big point), but another
> > > > important one is sharing. Without mmap(), an (real) shared library
> > > > support most likely will require special kernel support.
> > >
> > > What aspect of shared libraries are you aching for? Dynamic
> > > linking or the dynamic loading?
> >
> > 3rd: Sharing pages.
>
> segment(3) already provides this.

hmm, so segment(3)+segattach(2) can be seen as a kind of a frontend
for mmap() ;-)

But now I'm curious how executables and shared libraries are
actually handled on plan9.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Enrico Weigelt]

* Roman V. Shaposhnik <rvs@sun.com> wrote:
> On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
> > Convenience is one point (sometimes be a big point), but another
> > important one is sharing. Without mmap(), an (real) shared library
> > support most likely will require special kernel support.
>
> What aspect of shared libraries are you aching for? Dynamic
> linking or the dynamic loading?

3rd: Sharing pages.

Well, this perhaps also could be done if the kernel would be able
to detect equal pages and automatically map them together (maybe
w/ copy-on-write again).


BTW mmap() is also nice for creating shared memory between
(local) processes. For example RDBMS'es can get a huge benetit
from this.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Richard Miller]

> Ambric is
> pretty well aware of what the transputer did.

Glad to hear it - there were some good ideas behind the transputer which
are worth recycling.  From your description it sounds like the ambric has
some interesting new refinements as well.

Re: [9fans] mmap

[ron minnich]

On Sat, Aug 2, 2008 at 6:22 AM, Richard Miller <9fans@hamnavoe.com> wrote:
>> has hardware channels. And you can
>> call from channel
>> and execute code being sent down a channel to you from another cpu.
> ...
>> it's a very interesting architecture, to say the least. For me anyway
>> the most novel thing I've seen in a while.
>
> Interesting but not novel.  Remember the transputer, circa 1985?  Hardware
> channels, both inter- and intra-chip.  The normal way of loading code was
> to read it down a link from another processor - used for bootstrapping
> an array of transputers, but also could be used for dynamic propagation
> of processes through the network.
>

yes, but IIRC the transputer did not have the ability to execute code
from a channel in the way the ambric does -- literally having a PC
that includes a channel ID. There are a few other wrinkles to the
ambric that make it a little neater than the transputer. Ambric is
pretty well aware of what the transputer did.

rno

Re: [9fans] mmap

[Richard Miller]

> has hardware channels. And you can
> call from channel
> and execute code being sent down a channel to you from another cpu.
...
> it's a very interesting architecture, to say the least. For me anyway
> the most novel thing I've seen in a while.

Interesting but not novel.  Remember the transputer, circa 1985?  Hardware
channels, both inter- and intra-chip.  The normal way of loading code was
to read it down a link from another processor - used for bootstrapping
an array of transputers, but also could be used for dynamic propagation
of processes through the network.

Re: [9fans] mmap

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Thu, Jul 31, 2008 at 5:32 PM, ron minnich <rminnich@gmail.com> wrote:

> here is a thought:
>
> the kernel does mmap for code/data. This is because we think of a file
> as a segment of data that somehow maps well to a segment of memory.
>
> You wouldn't execute code from a stream, now, would you?
>
> Well, this: http://www.ambric.com/
>
> has hardware channels. And you can
> call from channel
> and execute code being sent down a channel to you from another cpu.


>
> There's no real analogue to this in any OS I've used for a while ...
>
> You could write the mcilroy sieve in this very directly. You could
> even, when starting a new thread, push the code down to the next cpu.
> And that cpu is paused in a call from channel until it gets its code.
> A nice way to keep them idle until you need them.
>
> it's a very interesting architecture, to say the least. For me anyway
> the most novel thing I've seen in a while.
>

The ARM 7 and ARM 9 procs in the Nintendo DS can talk back and forth via a
FIFO at a pretty low level :-)

http://www.double.co.nz/nintendo_ds/nds_develop7.html

Dave

>
> ron
>
>

[-- Attachment #2: Type: text/html, Size: 1931 bytes --]

Re: [9fans] mmap

[ron minnich]

here is a thought:

the kernel does mmap for code/data. This is because we think of a file
as a segment of data that somehow maps well to a segment of memory.

You wouldn't execute code from a stream, now, would you?

Well, this: http://www.ambric.com/

has hardware channels. And you can
call from channel
and execute code being sent down a channel to you from another cpu.

There's no real analogue to this in any OS I've used for a while ...

You could write the mcilroy sieve in this very directly. You could
even, when starting a new thread, push the code down to the next cpu.
And that cpu is paused in a call from channel until it gets its code.
A nice way to keep them idle until you need them.

it's a very interesting architecture, to say the least. For me anyway
the most novel thing I've seen in a while.

ron

Re: [9fans] mmap

[Bakul Shah]

On Thu, 31 Jul 2008 22:39:43 BST Charles Forsyth <forsyth@terzarima.net>  wrote:
> > Does anyone have pointers to *nice* interfaces for doing this?  I don't
> > care which OS, but I'd like to not have the implementation I'm
> > requesting suck because I missed some gotchas that someone else already hit.
>
> paul lalonde's request is a good one, but i think a useful qualification is
> that it should also be shown in practice to have a reasonable implementation
> on several agreeably different architectures.  one of the reasons mmap is
> bad is that like job control it whacked some vague ideas into an existing
> system without reference to its existing conventions, then vaguely extended
> it in a vague way from time to time, vaguely different from platform to
> platform, and with vague documentation.  ``could they be any more vague?''

I vaguely remember seeing a comment on Karels and McKusick's
original mmap proposal (about 20 years ago).  I think it was
by dmr.  Something to the effect that if such a facility is
desired it should be made as integral to the OS as an inode
is to Unix (and not bolted on the side). Makes for an
interesting thought experiment as potentially all sorts of
simplifications are possible.

Re: [9fans] mmap

[Charles Forsyth]

> Does anyone have pointers to *nice* interfaces for doing this?  I don't
> care which OS, but I'd like to not have the implementation I'm
> requesting suck because I missed some gotchas that someone else already hit.

paul lalonde's request is a good one, but i think a useful qualification
is that it should also be shown in practice to have a reasonable implementation
on several agreeably different architectures.  one of the reasons mmap is
bad is that like job control it whacked some vague ideas into an existing
system without reference to its existing conventions, then vaguely extended
it in a vague way from time to time, vaguely different from platform to platform,
and with vague documentation.  ``could they be any more vague?''

Re: [9fans] mmap

[Paweł Lasek]

On Wed, Jul 30, 2008 at 18:36, Roman V. Shaposhnik <rvs@sun.com> wrote:
> As Russ, quite rightfully, pointed out: mmap() means different things
> to different people. The tragic part is, that it tries to do lots of
> things but it doesn't do anything particularly well. Personally, my
> experience of trying to use mmap() as a useful abstraction for the
> CPU's MMU was the last straw. It can't do even that reliably
> and in a portable fashion. Not to digress, but I was even more surprised
> to learn that there's not a single API on UNIX that can:
>    http://thread.gmane.org/gmane.os.netbsd.devel.kernel/6392/focus=6457
>
> So, what mmap() (the way it is done on UNIX) is good for? Here's my
> personal list. Feel free to add (and suggest alternatives on systems
> lacking mmap() such as Plan9):
>   * a *lazy* way of handling highly irregular I/O over large files.
>     Cases, where you can't really tell which parts of the file are
>     going to be needed. The best example here is mmap() on exec.
>     You don't have to read() all of .text if the actual execution path
>     only takes you to a couple of routines.
>   * an optimization for regular I/O. To some extent, I've always
>     wondered why read always takes its second argument -- a lot of
>     times I don't really care where the buffer with the data I need
>     ends up in my address space.
>
> That's pretty much it. Everything else, feels like a hack in a dire
> need of a better abstraction.

SBCL uses it to kick out system vm out of managing it's memory. At
least the version I got
on linux calls mmap() with MAP_ANONYMOUS, rwx permissions and I think MAP_FIXED.

Unfortunately, when you disable overcommit it fails to start on my
machine, because I clearly don't have
8GB of free memory&swap for it (I total at 2.5 GB w/ swap). I wish
someone gave a plain interface to just
took control of memory space...

I also had seen a simple malloc() implementation (targeted at people
who want to write their own systems) that
had a simple form of mmap/unmap calls (without mapping of files) as
it's only requirement. If it's the only kernel-exported API for
managing memory, it can make sense. I think that was done in L4 API,
which organizes it's processes by address spaces (the distinction
between what we call a process and thread being only their address
spaces - a process was a set of threads grouped by address space). A
process could also implement it's own pager and create processes that
used it's address space (basically what 9vx does, as you could also
install appropriate syscall trampolines).

As a counter-example, MS Windows' Notepad.exe is one of the worst
offenders when it comes to mmap().
To put it simply, the reason it doesn't work on large files is that it
mmaps them. ALWAYS. It's even stated in IFS SDK,
that "implementation of memory mapping mechanism is required to get
notepad to work on the filesystem in question".

> Thanks,
> Roman.
>

-- 
Paweł Lasek

Re: [9fans] mmap

[Joel C. Salomon]

On Wed, Jul 30, 2008 at 12:25 PM, Joel C. Salomon
<joelcsalomon@gmail.com> wrote:
> I forget who said it,

Found it in <http://9fans.net/archive/2002/08/130>:

On Tue, 13 Aug 2002 07:43:45 -0400, David Gordon Hogan <dhog@pla...> wrote:
> > On freebsd and Linux, exec happens via an mmap (more or less).
<snip>
> So [gigantic leap here], not only does Linux have ~250 system
> calls, but most of them can be emulated with mmap?

--Joel

Re: [9fans] mmap

[erik quanstrom]

> > I forget who said it, and the local firewall won't allow me to search
> > the online copy of /sys/games/lib/fortunes, but there should be a line
> > there about Linux having 200+ system calls, most of them emulatable
> > with mmap().
> >
> what nonsense is that?

/sys/games/lib/fortunes:4184: A handful of characteristics of Unix are responsible for its resilience. First, Unix is simple: whereas some operating systems implement thousands of system calls and have unclear design goals, Unix systems typically implement only hundreds of system calls and have a very clear design. -- Linux Kernel Development, 2nd Ed. by Robert Love

- erik

Re: [9fans] mmap

[Kernel Panic]

Joel C. Salomon wrote:
> On Wed, Jul 30, 2008 at 11:29 AM, Enrico Weigelt <weigelt@metux.de> wrote:
>
>> Convenience is one point (sometimes be a big point), but another
>> important one is sharing. Without mmap(), an (real) shared library
>> support most likely will require special kernel support.
>>
>
> Actually, almost any kernel support for shared libraries will need
> something like mmap() internally.
>
> I forget who said it, and the local firewall won't allow me to search
> the online copy of /sys/games/lib/fortunes, but there should be a line
> there about Linux having 200+ system calls, most of them emulatable
> with mmap().
>
what nonsense is that?
> --Joel
>

cinap

Re: [9fans] mmap

[Paul Lalonde]

Roman V. Shaposhnik wrote:
> Personally, my
> experience of trying to use mmap() as a useful abstraction for the
> CPU's MMU was the last straw. It can't do even that reliably
> and in a portable fashion. Not to digress, but I was even more surprised
> to learn that there's not a single API on UNIX that can:
>     http://thread.gmane.org/gmane.os.netbsd.devel.kernel/6392/focus=6457
>

Amen.  I've had to get our OS team to shoehorn in some remap() calls
that let me point two(many) virtual addresses to a common physical
address.  It's disgusting that the process for doing this with mmap
requires /proc/#/mem and other scary things.

Does anyone have pointers to *nice* interfaces for doing this?  I don't
care which OS, but I'd like to not have the implementation I'm
requesting suck because I missed some gotchas that someone else already hit.

Thanks,
    Paul

Re: [9fans] mmap

[Roman V. Shaposhnik]

On Tue, 2008-07-29 at 12:28 -0400, Venkatesh Srinivas wrote:
> As far as interfaces go, mmap() is pretty tragic - the underlying
> translation structures can express more interesting things, some of
> which are even worth doing.

I can't agree more. The way I look at it is that mmap() seems to
be the answer but nobody ever bothered to ask the question it is
supposed to answer.

> In a system like Plan 9, where your file servers are on the other side
> of a 9P link, this mmap thing seems dubious. If what you want is the
> convenience that you get from having all the bytes in memory, reading
> them all in wouldn't be too hard. mmap()s magic really arises when you
> have a page-cache-like-thing.

As Russ, quite rightfully, pointed out: mmap() means different things
to different people. The tragic part is, that it tries to do lots of
things but it doesn't do anything particularly well. Personally, my
experience of trying to use mmap() as a useful abstraction for the
CPU's MMU was the last straw. It can't do even that reliably
and in a portable fashion. Not to digress, but I was even more surprised
to learn that there's not a single API on UNIX that can:
    http://thread.gmane.org/gmane.os.netbsd.devel.kernel/6392/focus=6457

So, what mmap() (the way it is done on UNIX) is good for? Here's my
personal list. Feel free to add (and suggest alternatives on systems
lacking mmap() such as Plan9):
   * a *lazy* way of handling highly irregular I/O over large files.
     Cases, where you can't really tell which parts of the file are
     going to be needed. The best example here is mmap() on exec.
     You don't have to read() all of .text if the actual execution path
     only takes you to a couple of routines.
   * an optimization for regular I/O. To some extent, I've always
     wondered why read always takes its second argument -- a lot of
     times I don't really care where the buffer with the data I need
     ends up in my address space.

That's pretty much it. Everything else, feels like a hack in a dire
need of a better abstraction.

Thanks,
Roman.

Re: [9fans] mmap

[Joel C. Salomon]

On Wed, Jul 30, 2008 at 11:29 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> Convenience is one point (sometimes be a big point), but another
> important one is sharing. Without mmap(), an (real) shared library
> support most likely will require special kernel support.

Actually, almost any kernel support for shared libraries will need
something like mmap() internally.

I forget who said it, and the local firewall won't allow me to search
the online copy of /sys/games/lib/fortunes, but there should be a line
there about Linux having 200+ system calls, most of them emulatable
with mmap().

--Joel

Re: [9fans] mmap

[Roman V. Shaposhnik]

On Wed, 2008-07-30 at 17:29 +0200, Enrico Weigelt wrote:
> Convenience is one point (sometimes be a big point), but another
> important one is sharing. Without mmap(), an (real) shared library
> support most likely will require special kernel support.

What aspect of shared libraries are you aching for? Dynamic
linking or the dynamic loading?

Thanks,
Roman.

Re: [9fans] mmap

[Enrico Weigelt]

* Venkatesh Srinivas <me@acm.jhu.edu> wrote:

... redirecting back to 9fans ;-P

> As far as interfaces go, mmap() is pretty tragic - the underlying
> translation structures can express more interesting things, some of
> which are even worth doing.

Well, the biggest problem, IMHO are the incompatible semantics
between platforms, and some things IMHO aren't needed at all.

What I expect from an clean mmap() is:

* map in a given region of some file into process's address space
* mapping types:
    a) readonly
    b) readwrite
    c) private / copy-on-write
* either let the app or the kernel decide on start address
* optional hints on intended access patterns (eg. if it might
  be wise to do read-ahead)

Ah, and there should be a special (size-limited) kernel device
which holds evrything in RAM exclusively (never get swapped out)
for holding critical security information.

> There have even been OSes that let userland apps play with their address
> spaces in far more interesting ways - KeyKOS and EROS come to mind. And
> they were even fast, or something.

hmm, never heared about them. what are they doing at this point ?
>
> In a system like Plan 9, where your file servers are on the other
> side of a 9P link, this mmap thing seems dubious. If what you want is the
> convenience that you get from having all the bytes in memory, reading
> them all in wouldn't be too hard. mmap()s magic really arises when you
> have a page-cache-like-thing.

Well, mmap() is a tricky thing since it breaks common filesystem
semantics. So it can only be done, and only makes sense on specific
kind of files/devices which are only acessed block-wise. So the
absolute MUST is that the underlying file supports (at least block-
wise) random access - mmap()'ed streams are completely nonsense.

Convenience is one point (sometimes be a big point), but another
important one is sharing. Without mmap(), an (real) shared library
support most likely will require special kernel support.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[Charles Forsyth]

> i think that's right, and that's the interesting case to investigate

provided, of course, that you're interested in the applications that might use it.
otherwise it will just complicate things to no good effect.

Re: [9fans] mmap

[Charles Forsyth]

> As far as interfaces go, mmap() is pretty tragic - the underlying
> translation structures can express more interesting things, some of
> which are even worth doing.

> There have even been OSes that let userland apps play with their address
> spaces in far more interesting ways

i think that's right, and that's the interesting case to investigate

Re: [9fans] mmap

[Venkatesh Srinivas]

As far as interfaces go, mmap() is pretty tragic - the underlying
translation structures can express more interesting things, some of
which are even worth doing.

There have even been OSes that let userland apps play with their address
spaces in far more interesting ways - KeyKOS and EROS come to mind. And
they were even fast, or something.

In a system like Plan 9, where your file servers are on the other side
of a 9P link, this mmap thing seems dubious. If what you want is the
convenience that you get from having all the bytes in memory, reading
them all in wouldn't be too hard. mmap()s magic really arises when you
have a page-cache-like-thing.

--vs

Re: [9fans] mmap

[Roman V. Shaposhnik]

ron minnich wrote:
> On Tue, Jul 29, 2008 at 8:19 AM, erik quanstrom <quanstro@coraid.com> wrote:
>
>
>> you can't make the assumption that a file is local in *ix, either.
>>
>>
>
> in fact, for the last 20 years, every program run on a sunos/solaris
> machine has used mmap for the exec.
>
mmap() is everywhere on Solaris. Even when you do something as trivial
as sort(1) --
it'll try to mmap() the operands and only if that fails it is going to
use read()/write().

Thanks,
Roman.

P.S. Which is not that bad of an optimization if you ask me...

Re: [9fans] mmap

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1293 bytes --]

On Tue, Jul 29, 2008 at 8:04 AM, Alexander Sychev <santucco@gmail.com>wrote:

> On Tue, 29 Jul 2008 12:52:14 +0400, Enrico Weigelt <weigelt@metux.de>
> wrote:
>
> Hi!
>
>  an full mmap() is a really nice thing. It can make a lot things
>> easier if you just map the whole file into the process' memory and
>> let the kernel handle the actual IO.
>>
>
> Yes, it is comfortable. But just think a bit - what will you do in the mmap
> implementation when you had mapped a remote file (in Plan9 you can't be sure
> some file is local or it is really just a file), and the connection has just
> been broken? Surprise!


>
> For example, in Windows you will have an "access violation". It is very
> funny to rewrite a code for using the good old file i/o instead of mmap.
> In Linux you can't map a non-regular file.
>
> Yes, you can read entire the file into the memory, but a size of the file
> can be very big and you will destroy the main advantage of mmap - the
> reading of only needed pieces of the file, not entire.
>

And if you read it in in pieces, how can you be sure the file isn't changing
on the system it's hosted on while you're looking at it?


>
> I beleive, mmap is not for distributed systems.
>
> --
> Best regards,
>        santucco
>
>

[-- Attachment #2: Type: text/html, Size: 2128 bytes --]

Re: [9fans] mmap

[ron minnich]

On Tue, Jul 29, 2008 at 8:19 AM, erik quanstrom <quanstro@coraid.com> wrote:

> you can't make the assumption that a file is local in *ix, either.
>

in fact, for the last 20 years, every program run on a sunos/solaris
machine has used mmap for the exec.

ron

Re: [9fans] mmap

[erik quanstrom]

> Yes, it is comfortable.

where's jim when you need him?

> But just think a bit - what will you do in the
> mmap implementation when you had mapped a remote file (in Plan9 you can't
> be sure some file is local or it is really just a file), and the
> connection has just been broken? Surprise!

you can't make the assumption that a file is local in *ix, either.

- erik

Re: [9fans] mmap

[Alexander Sychev]

On Tue, 29 Jul 2008 12:52:14 +0400, Enrico Weigelt <weigelt@metux.de>
wrote:

Hi!

> an full mmap() is a really nice thing. It can make a lot things
> easier if you just map the whole file into the process' memory and
> let the kernel handle the actual IO.

Yes, it is comfortable. But just think a bit - what will you do in the
mmap implementation when you had mapped a remote file (in Plan9 you can't
be sure some file is local or it is really just a file), and the
connection has just been broken? Surprise!

For example, in Windows you will have an "access violation". It is very
funny to rewrite a code for using the good old file i/o instead of mmap.
In Linux you can't map a non-regular file.

Yes, you can read entire the file into the memory, but a size of the file
can be very big and you will destroy the main advantage of mmap - the
reading of only needed pieces of the file, not entire.

I beleive, mmap is not for distributed systems.

--
Best regards,
	santucco

Re: [9fans] mmap

[Charles Forsyth]

> It can make a lot things
> easier if you just map the whole file into the process' memory and
> let the kernel handle the actual IO.

the word "superficially" should be in there somewhere.

Re: [9fans] mmap

[Enrico Weigelt]

* Russ Cox <rsc@swtch.com> wrote:

Hi,

<big_snip />

I don't know much of native plan9 (only using plan9port), but IMHO
an full mmap() is a really nice thing. It can make a lot things
easier if you just map the whole file into the process' memory and
let the kernel handle the actual IO.

Some time ago I hacked up a little XML-based database (just to
show myself that it needn't to be that ugly slow as Xindice ;-)),
and being able to conveniently work with pointers instead of ugly
seek() made really fun ;-P


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] mmap

[erik quanstrom]

> in a traditional full-featured mmap

i've noticed that some combinations of words
are scarier than others.

☺

- erik