[9fans] Memory management questions/not plan-9 specific

[9fans] Memory management questions/not plan-9 specific

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have an application I'm building which requires OS support to allow  
a user-space function to fill a page on page-faults.  Ideally, I  
could reserve a chunk of address space but not back it with memory,  
and then on fault my handler would serve out data from some small  
cache of user-managed physical pages.

My google-fu has been weak in finding such a system-level API in any  
OS.  Has this got a name I should be searching on?  I can't believe  
no-one has implemented user-level page replacement.

Thanks,
    Paul

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFHMKM4pJeHo/Fbu1wRAgezAKCSDVI812jisKkliXjpsWTMU3AlogCcDkOl
liIiMj9Y/2xwHx1iFKjSZ8w=
=CAPO
-----END PGP SIGNATURE-----

Re: [9fans] Memory management questions/not plan-9 specific

[Philippe Anel]

l4 (http://en.wikipedia.org/wiki/L4_microkernel_family) has such api ... 
but it is only a kernel.

Paul Lalonde a écrit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have an application I'm building which requires OS support to allow 
> a user-space function to fill a page on page-faults.  Ideally, I could 
> reserve a chunk of address space but not back it with memory, and then 
> on fault my handler would serve out data from some small cache of 
> user-managed physical pages.
>
> My google-fu has been weak in finding such a system-level API in any 
> OS.  Has this got a name I should be searching on?  I can't believe 
> no-one has implemented user-level page replacement.
>
> Thanks,
>    Paul
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.3 (Darwin)
>
> iD8DBQFHMKM4pJeHo/Fbu1wRAgezAKCSDVI812jisKkliXjpsWTMU3AlogCcDkOl
> liIiMj9Y/2xwHx1iFKjSZ8w=
> =CAPO
> -----END PGP SIGNATURE-----
>
>

Re: [9fans] Memory management questions/not plan-9 specific

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Nov 6, 2007, at 10:29 AM, Robert William Fuller wrote:
>> In UNIX, set up a signal handler for SIGSEGV.

Not quite enough - I still need to reserve some address space.  mmap  
doesn't let me reserve it without backing it.

Paul

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFHMKjEpJeHo/Fbu1wRAmoFAJsFbl4KxIss+zqcJtq4BHv/eja3UACfbw/i
kVYQXtXGl667sAYBmehus9k=
=6hML
-----END PGP SIGNATURE-----

Re: [9fans] Memory management questions/not plan-9 specific

[Philippe Anel]

You also can do this in windows :


#include <windows.h>
#include <stdio.h>

void sysfatal(char * reason)
{
    printf("'%s' failed with error %d\n", reason, GetLastError());
    ExitProcess(0);
}

PUCHAR    Page;

int filter(unsigned int code, struct _EXCEPTION_POINTERS *ep)
{
    if (code == EXCEPTION_ACCESS_VIOLATION) {
        if(ep->ExceptionRecord->ExceptionInformation[1] == (DWORD_PTR) 
Page){
            Page = VirtualAlloc(Page, 4096, MEM_COMMIT, PAGE_READWRITE);
            if(Page == 0)
                sysfatal("VirtualAlloc: commit");
            return EXCEPTION_CONTINUE_EXECUTION;
        }
        return EXCEPTION_EXECUTE_HANDLER;
    }
    return EXCEPTION_CONTINUE_SEARCH;
}

int main(int argc, char ** argv)
{
    // Reserve
    Page = VirtualAlloc(0, 4096, MEM_RESERVE, PAGE_READWRITE);
    if(Page == 0)
        sysfatal("VirtualAlloc: reserve");
    __try{
        // Touch
        *Page = 0;
        puts("touched.");
    }
    __except(filter(GetExceptionCode(), GetExceptionInformation()))
    {
        puts("in except");
    }
    return 0;
}


Philippe Anel a écrit :
> l4 (http://en.wikipedia.org/wiki/L4_microkernel_family) has such api 
> ... but it is only a kernel.
>
> Paul Lalonde a écrit :
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I have an application I'm building which requires OS support to allow 
>> a user-space function to fill a page on page-faults.  Ideally, I 
>> could reserve a chunk of address space but not back it with memory, 
>> and then on fault my handler would serve out data from some small 
>> cache of user-managed physical pages.
>>
>> My google-fu has been weak in finding such a system-level API in any 
>> OS.  Has this got a name I should be searching on?  I can't believe 
>> no-one has implemented user-level page replacement.
>>
>> Thanks,
>>    Paul
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.3 (Darwin)
>>
>> iD8DBQFHMKM4pJeHo/Fbu1wRAgezAKCSDVI812jisKkliXjpsWTMU3AlogCcDkOl
>> liIiMj9Y/2xwHx1iFKjSZ8w=
>> =CAPO
>> -----END PGP SIGNATURE-----
>>
>>
>
>
>

Re: [9fans] Memory management questions/not plan-9 specific

[ron minnich]

On 11/6/07, Paul Lalonde <plalonde@telus.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I have an application I'm building which requires OS support to allow
> a user-space function to fill a page on page-faults.  Ideally, I
> could reserve a chunk of address space but not back it with memory,
> and then on fault my handler would serve out data from some small
> cache of user-managed physical pages.
>
> My google-fu has been weak in finding such a system-level API in any
> OS.  Has this got a name I should be searching on?  I can't believe
> no-one has implemented user-level page replacement.


you can look at the code in this:
http://mbgokhale.org/rminnich/job/zounds/zx.tar

docs are at : http://mbgokhale.org/rminnich/job/zounds/zounds.ps

i still get the occasional note from someone who is using it.

ron

Re: [9fans] Memory management questions/not plan-9 specific

[ron minnich]

On 11/6/07, Robert William Fuller <hydrologiccycle@gmail.com> wrote:
> Paul Lalonde wrote:
> >
> > On Nov 6, 2007, at 10:29 AM, Robert William Fuller wrote:
> >>> In UNIX, set up a signal handler for SIGSEGV.
> >
> > Not quite enough - I still need to reserve some address space.  mmap
> > doesn't let me reserve it without backing it.
>
> It does by default on Linux :-)
>

mmap /dev/zero over the region you want. Or not, you can just any
piece of your address space, actually.

mprotect it (page-aligned, now!0 so you can't access it. record the
fact that you did this. i.e. set up data structures that allow you to
tell, in the segv handler, that it's a "page fault" and not a true
screwup in your code.

in the segv handler, get the fault address and type, search your
structs, figure out what to do. Be sure to compute page frame from the
address, etc. etc. fix it up with memprotect. record what you did.
return.

that's pretty much what I did.

ron

Re: [9fans] Memory management questions/not plan-9 specific

[Robert William Fuller]

Paul Lalonde wrote:
> I have an application I'm building which requires OS support to allow a
> user-space function to fill a page on page-faults.  Ideally, I could
> reserve a chunk of address space but not back it with memory, and then
> on fault my handler would serve out data from some small cache of
> user-managed physical pages.
> 
> My google-fu has been weak in finding such a system-level API in any
> OS.  Has this got a name I should be searching on?  I can't believe
> no-one has implemented user-level page replacement.

In UNIX, set up a signal handler for SIGSEGV.

Re: [9fans] Memory management questions/not plan-9 specific

[Robert William Fuller]

Paul Lalonde wrote:
> 
> On Nov 6, 2007, at 10:29 AM, Robert William Fuller wrote:
>>> In UNIX, set up a signal handler for SIGSEGV.
> 
> Not quite enough - I still need to reserve some address space.  mmap
> doesn't let me reserve it without backing it.

It does by default on Linux :-)

Re: [9fans] Memory management questions/not plan-9 specific

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ah, I see the light.

I'll give it a shot :-)

Paul

On Nov 6, 2007, at 10:28 AM, ron minnich wrote:

> On 11/6/07, Robert William Fuller <hydrologiccycle@gmail.com> wrote:
>> Paul Lalonde wrote:
>>>
>>> On Nov 6, 2007, at 10:29 AM, Robert William Fuller wrote:
>>>>> In UNIX, set up a signal handler for SIGSEGV.
>>>
>>> Not quite enough - I still need to reserve some address space.  mmap
>>> doesn't let me reserve it without backing it.
>>
>> It does by default on Linux :-)
>>
>
> mmap /dev/zero over the region you want. Or not, you can just any
> piece of your address space, actually.
>
> mprotect it (page-aligned, now!0 so you can't access it. record the
> fact that you did this. i.e. set up data structures that allow you to
> tell, in the segv handler, that it's a "page fault" and not a true
> screwup in your code.
>
> in the segv handler, get the fault address and type, search your
> structs, figure out what to do. Be sure to compute page frame from the
> address, etc. etc. fix it up with memprotect. record what you did.
> return.
>
> that's pretty much what I did.
>
> ron

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFHMMqypJeHo/Fbu1wRAqUQAJ4hn3Di6umwfmDRkz2l9vGoxlAoZACgsXgc
ejFMKTf4qP4qJLIOPkzcwMQ=
=HC5E
-----END PGP SIGNATURE-----

Re: [9fans] Memory management questions/not plan-9 specific

[Dave Eckhardt]

> My google-fu has been weak in finding such a system-level
> API in any OS.

The Mach name for this was "external pager", see for example
"Extending the Mach External Pager Interface to Accommodate
User-Level Page Replacement Policies" (1990).

Dave Eckhardt