Re: [9fans] plan 9 overcommits memory?

Re: [9fans] plan 9 overcommits memory?

[geoff]

If system calls were the only way to change memory allocation, one
could probably keep a strict accounting of pages allocated and fail
system calls that require more VM than is available.  But neither Plan
9 nor Unix works that way.  The big exception is stack growth.  The
kernel automatically extends a process's stack segment as needed.  On
the pc, Plan 9 currently limits user-mode stacks to 16MB.  On a CPU
server with 200 processes (fairly typical), that's 3.2GB of VM one
would have to commit just for stacks.  With 2,000 processes, that
would rise to 32GB just for stacks.

Re: [9fans] plan 9 overcommits memory?

[Scott Schwartz]

On Sun, Sep 02, 2007 at 11:38:44PM -0400, geoff@plan9.bell-labs.com wrote:
> would have to commit just for stacks.  With 2,000 processes, that
> would rise to 32GB just for stacks.

With 4GB RAM, wouldn't you allocate at least that much swap
no matter what?

Re: [9fans] plan 9 overcommits memory?

[Uriel]

Except that swap, is, as far as I have been able to figure out, broken.

uriel

On 3 Sep 2007 01:35:14 -0400, Scott Schwartz <schwartz@bio.cse.psu.edu> wrote:
> On Sun, Sep 02, 2007 at 11:38:44PM -0400, geoff@plan9.bell-labs.com wrote:
> > would have to commit just for stacks.  With 2,000 processes, that
> > would rise to 32GB just for stacks.
>
> With 4GB RAM, wouldn't you allocate at least that much swap
> no matter what?

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> If system calls were the only way to change memory allocation, one
> could probably keep a strict accounting of pages allocated and fail
> system calls that require more VM than is available.  But neither Plan
> 9 nor Unix works that way.  The big exception is stack growth.  The
> kernel automatically extends a process's stack segment as needed.  On
> the pc, Plan 9 currently limits user-mode stacks to 16MB.  On a CPU
> server with 200 processes (fairly typical), that's 3.2GB of VM one
> would have to commit just for stacks.  With 2,000 processes, that
> would rise to 32GB just for stacks.

16MB for stacks seems awful high to me.  are there any programs that
need even 1/32th of that?  512k is still 32k levels of recursion of
a function needing 4 long arguments.  a quick count on my home machine
and some coraid servers don't show any processes using more than 1
page of stack.

doing strict accounting on the pages allocated i think would be an
improvement.  i also don't see a reason not to shrink the maximum
stack size.

the current behavior seems pretty exploitable to me.  even remotely,
if one can force stack/brk allocation via smtp, telnet or whatnot.

- erik

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

>> would have to commit just for stacks.  With 2,000 processes, that
>> would rise to 32GB just for stacks.
>
> With 4GB RAM, wouldn't you allocate at least that much swap
> no matter what?

that's pretty expensive if you're booting from flash and not using a remote
fileserver.  8GB flash is expensive, not to mention deadly slow.

also, why should i have to have swap?  i really don't want it.  it
introduces new failure modes and could introduce wide latency
variations.  linux called, it wants it's choppy, laggy ui back.

- erik

Re: [9fans] plan 9 overcommits memory?

[Sape Mullender]

>> If system calls were the only way to change memory allocation, one
>> could probably keep a strict accounting of pages allocated and fail
>> system calls that require more VM than is available.  But neither Plan
>> 9 nor Unix works that way.  The big exception is stack growth.  The
>> kernel automatically extends a process's stack segment as needed.  On
>> the pc, Plan 9 currently limits user-mode stacks to 16MB.  On a CPU
>> server with 200 processes (fairly typical), that's 3.2GB of VM one
>> would have to commit just for stacks.  With 2,000 processes, that
>> would rise to 32GB just for stacks.
>
> 16MB for stacks seems awful high to me.  are there any programs that
> need even 1/32th of that?  512k is still 32k levels of recursion of
> a function needing 4 long arguments.  a quick count on my home machine
> and some coraid servers don't show any processes using more than 1
> page of stack.
>
> doing strict accounting on the pages allocated i think would be an
> improvement.  i also don't see a reason not to shrink the maximum
> stack size.
>
> the current behavior seems pretty exploitable to me.  even remotely,
> if one can force stack/brk allocation via smtp, telnet or whatnot.
>
> - erik

Most applications probably use much less than 1 MB, but a lot depends
on who wrote the program.  Our threaded programs typically have a 4K
or 8K (K, not M) fixed-size stack per thread and that works fine,
although you have to remember not to declare big arrays/structs as
local variables.  malloc and free become good friends in threaded
programs.

As to guarantees that you won't run our of memory, they're almost
impossible to give.  Programmer generally don't know how much memory
their applications will use, so they can't reasonably preallocate.

You see the same thing with real time.  Nobody knows how much time
each task will consume beforehand.

It would be cool to be able to get a handle on being able to shrink
the memory occupied by an application dynamically.  Malloc (through
brk()) grows the memory footprint, but free does not shrink it.
The same is true for the stack.  Once allocated, it doesn't get freed
until the process exits.

	Sape

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> Most applications probably use much less than 1 MB, but a lot depends
> on who wrote the program.  Our threaded programs typically have a 4K
> or 8K (K, not M) fixed-size stack per thread and that works fine,
> although you have to remember not to declare big arrays/structs as
> local variables.  malloc and free become good friends in threaded
> programs.

> As to guarantees that you won't run our of memory, they're almost
> impossible to give.  Programmer generally don't know how much memory
> their applications will use, so they can't reasonably preallocate.

that's a much stronger condition than "if there isn't backing memory,
brk fails".  perhaps that is difficult.  even if the actual condition
is estimated, wouldn't that be an improvement.

perhaps one could reserve, say 16MB total for stack space.  (or maybe
some percentage of physical memory.)  this could eliminate overcommits
for brk'ed memory.

> You see the same thing with real time.  Nobody knows how much time
> each task will consume beforehand.

> It would be cool to be able to get a handle on being able to shrink
> the memory occupied by an application dynamically.  Malloc (through
> brk()) grows the memory footprint, but free does not shrink it.
> The same is true for the stack.  Once allocated, it doesn't get freed
> until the process exits.

yes it would.  does plan 9 have programs that could make use of this
currently?

- erik

Re: [9fans] plan 9 overcommits memory?

[Sape Mullender]

>> It would be cool to be able to get a handle on being able to shrink
>> the memory occupied by an application dynamically.  Malloc (through
>> brk()) grows the memory footprint, but free does not shrink it.
>> The same is true for the stack.  Once allocated, it doesn't get freed
>> until the process exits.
>
> yes it would.  does plan 9 have programs that could make use of this
> currently?

No, and it would be hard to do it because you'd need ways to compact
fragmented memory after a lot of mallocs and frees.  And then, you'd
need a way to fix the pointers after compacting.

	Sape

Re: [9fans] plan 9 overcommits memory?

[john]

quanstro@quanstro.net wrote:
> also, why should i have to have swap?  i really don't want it.  it
> introduces new failure modes and could introduce wide latency
> variations.  linux called, it wants it's choppy, laggy ui back.
>

Also, it's broken, broken, broken on Plan 9 and nobody wants to fix it.
The upside to this is that we can just say how we don't want it anyway,
there's no conceivable reason anyone would want swap,  and operating
systems with working swap suck ;)


John "Has a Swap Partition and Doesn't Know Why" Floren

Re: [9fans] plan 9 overcommits memory?

[Gorka Guardiola]

On 9/3/07, john@csplan9.rit.edu <john@csplan9.rit.edu> wrote:
>
> Also, it's broken, broken, broken on Plan 9 and nobody wants to fix it.
> The upside to this is that we can just say how we don't want it anyway,
> there's no conceivable reason anyone would want swap,  and operating
> systems with working swap suck ;)
>
>
> John "Has a Swap Partition and Doesn't Know Why" Floren
>

Isn´t it more like John "wants somebody else to fix his swap instead of doing
himself" Floren?. If you think something is broken, fix it instead of
complaining.
If noone else likes it, at least you have your problem solved...

-- 
- curiosity sKilled the cat

Re: [9fans] plan 9 overcommits memory?

[Francisco J Ballesteros]

Fixing something nobody uses might be fun, but probably it's low on the stacks
of most people because we don't consume too much memory.

On 9/3/07, Gorka Guardiola <paurea@gmail.com> wrote:
> On 9/3/07, john@csplan9.rit.edu <john@csplan9.rit.edu> wrote:
> >
> > Also, it's broken, broken, broken on Plan 9 and nobody wants to fix it.
> > The upside to this is that we can just say how we don't want it anyway,
> > there's no conceivable reason anyone would want swap,  and operating
> > systems with working swap suck ;)
> >
> >
> > John "Has a Swap Partition and Doesn't Know Why" Floren
> >
>
> Isn´t it more like John "wants somebody else to fix his swap instead of doing
> himself" Floren?. If you think something is broken, fix it instead of
> complaining.
> If noone else likes it, at least you have your problem solved...
>
> --
> - curiosity sKilled the cat
>

Re: [9fans] plan 9 overcommits memory?

[john]

> On 9/3/07, john@csplan9.rit.edu <john@csplan9.rit.edu> wrote:
>>
>> Also, it's broken, broken, broken on Plan 9 and nobody wants to fix it.
>> The upside to this is that we can just say how we don't want it anyway,
>> there's no conceivable reason anyone would want swap,  and operating
>> systems with working swap suck ;)
>>
>>
>> John "Has a Swap Partition and Doesn't Know Why" Floren
>>
> 
> Isn´t it more like John "wants somebody else to fix his swap instead of doing
> himself" Floren?. If you think something is broken, fix it instead of
> complaining.
> If noone else likes it, at least you have your problem solved...
> 

I don't actually need the swap partition, it's just there... ummm... not
sure why; I installed on this machine before I found out that swap is 
broken. And it's not that I *think* swap is broken; it's been confirmed 
by others. If I ever dig up a really old laptop with 32 MB of RAM or
something, it could be worth it to try fixing swap, but since that itch
doesn't exist I'm not going to scratch it.

John

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> Also, it's broken, broken, broken on Plan 9

but could you describe what antisocial behavior it exhibits and how one
could reproduce this behavior?  i have never used to-disk paging on plan 9,
so i don't know.

> and nobody wants to fix it.

this has been a good discussion so far.  let's not go off in a bad direction.

> The upside to this is that we can just say how we don't want it anyway,
> there's no conceivable reason anyone would want swap,  and operating
> systems with working swap suck ;)

not sure how to parse this.  is there a particular case where you need to-disk
paging?  i don't see the use of to-disk paging.  perhaps my vision is limited.

in the one case where i might find it useful  -- in embedded systems, there's
typically more ram than persistant storage, so paging to "disk" makes no sense.

- erik

Re: [9fans] plan 9 overcommits memory?

[Francisco J Ballesteros]

> but could you describe what antisocial behavior it exhibits and how one
> could reproduce this behavior?  i have never used to-disk paging on plan 9,
> so i don't know.


Last time I tried the machine did freeze, like rock solid. It happen at some
point after the swap partition was being used (saw its usage increase in stats).
Not always the same time interval from hitting return (to consume
memory) and get
the thing frozen. But this was several years ago, IIRC.

Re: [9fans] plan 9 overcommits memory?

[john]

>> Also, it's broken, broken, broken on Plan 9
>
> but could you describe what antisocial behavior it exhibits and how one
> could reproduce this behavior?  i have never used to-disk paging on plan 9,
> so i don't know.
>

Well, when I used it on an old 32 MB laptop (terminal) and a 64 MB
desktop (cpu server), swap would seem to work all right until you
hit about 30-40% usage. This was the case with both systems; when
I asked about it, a couple other people mentioned the same behavior.
The thing is, it's pretty hard to test swap under normal usage; the only
time I ran into this problem was while compiling a new kernel.

>> and nobody wants to fix it.
>
> this has been a good discussion so far.  let's not go off in a bad direction.

I was just noting that when it has previously come up, the general
consensus is that nobody wants to fix it, which is actually pretty
reasonable--I'm guessing, as has been mentioned before, that the
number of people who could potentially need/want swap is very low,
especially since memory for older computers seems to grow on trees
(around here at least).

>
>> The upside to this is that we can just say how we don't want it anyway,
>> there's no conceivable reason anyone would want swap,  and operating
>> systems with working swap suck ;)
>
> not sure how to parse this.  is there a particular case where you need to-disk
> paging?  i don't see the use of to-disk paging.  perhaps my vision is limited.
>
> in the one case where i might find it useful  -- in embedded systems, there's
> typically more ram than persistant storage, so paging to "disk" makes no sense.
>

It's primarily old systems, I think, like that old laptop which wasn't worth
finding more RAM for. When I set up this shiny high-spec cpu server,
I let it put in swap space "just in case", but a couple users barely put
a dent in that, so it will probably never be used.

Re: [9fans] plan 9 overcommits memory?

[john]

>>> Also, it's broken, broken, broken on Plan 9
>>
>> but could you describe what antisocial behavior it exhibits and how one
>> could reproduce this behavior?  i have never used to-disk paging on plan 9,
>> so i don't know.
>>
>
> Well, when I used it on an old 32 MB laptop (terminal) and a 64 MB
> desktop (cpu server), swap would seem to work all right until you
> hit about 30-40% usage. This was the case with both systems; when
> I asked about it, a couple other people mentioned the same behavior.
> The thing is, it's pretty hard to test swap under normal usage; the only
> time I ran into this problem was while compiling a new kernel.
>

I forgot to write what happened when swap broke--like Nemo, I found
that the machine would lock solid, requiring a reboot.

John

Re: [9fans] plan 9 overcommits memory?

[geoff]

One might allocate at least 3.2GB of swap for a 4GB machine, but many
of our machines run with no swap, and we're probably not alone.  And
200 processes are not a lot.  Would you really have over 32GB of swap
allocated for a 4GB machine with 2,000 processes?

Programs can use a surprising amount of stack space.  A recent notable
example is venti/copy when copying from a nightly fossil dump score.
I think we want to be generous about maximum stack sizes.

I don't think that estimates of VM usage would be an improvement.  If
we can't get it exactly right, there will always be complaints.

Re: [9fans] plan 9 overcommits memory?

[Uriel]

This has been discussed before, but given how hard it is to debug
this, and that nobody seems to have enough interest/motivation to do
so, wouldn't it make more sense to totally remove swap support? I
really can't understand why things like il are removed when they
actually work, and swap is left alone, when it is actually broken.

The current swap just frustrates people who expect it to work, and
then have their systems freeze randomly. Maybe by disabling/remove
swap support, then if someone really needs swap he will fix it first
and then we can add it back.

Best wishes

uriel

On 9/3/07, Gorka Guardiola <paurea@gmail.com> wrote:
> On 9/3/07, john@csplan9.rit.edu <john@csplan9.rit.edu> wrote:
> >
> > Also, it's broken, broken, broken on Plan 9 and nobody wants to fix it.
> > The upside to this is that we can just say how we don't want it anyway,
> > there's no conceivable reason anyone would want swap,  and operating
> > systems with working swap suck ;)
> >
> >
> > John "Has a Swap Partition and Doesn't Know Why" Floren
> >
>
> Isn´t it more like John "wants somebody else to fix his swap instead of doing
> himself" Floren?. If you think something is broken, fix it instead of
> complaining.
> If noone else likes it, at least you have your problem solved...
>
> --
> - curiosity sKilled the cat
>

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> I don't actually need the swap partition, it's just there... ummm... not
> sure why; I installed on this machine before I found out that swap is
> broken. And it's not that I *think* swap is broken; it's been confirmed

it worked adequately to cover minor shortfalls in memory, which could happen
in the best of machines.  now there is typically so much physical memory
it hardly ever is invoked on my systems, unless i'm burning a CD from ramfs
and get the numbers wrong.

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

>> Well, when I used it on an old 32 MB laptop (terminal) and a 64 MB
>> desktop (cpu server), swap would seem to work all right until you
>> hit about 30-40% usage. This was the case with both systems; when
>> I asked about it, a couple other people mentioned the same behavior.
>> The thing is, it's pretty hard to test swap under normal usage; the only
>> time I ran into this problem was while compiling a new kernel.
>>
>
> I forgot to write what happened when swap broke--like Nemo, I found
> that the machine would lock solid, requiring a reboot.

years ago i would compile and link kernels on a 4mbyte 386-16/sx (really! and using cpu -c to run
awk, because there wasn't a 387).  i was in the same room as the file server.
you could tell when it was paging, which had a distinctive, dramatic sound.
it paged frequently when linking a kernel.  it survived.
if it's broken now, it sounds as though something changed that probably could
be tracked down and repaired.  (i tend to suspect the presence of notes,
including alarms, but that's just a suspicion, because of the interruptions in the kernel).
why bother?  perhaps the underlying cause is messing up something else too.
could use a useful simple test case, though.  ideally, without graphics.

Re: [9fans] plan 9 overcommits memory?

[Uriel]

Can't find out for sure until someone fixes it :))

One more reason IMHO why we are best off having it disabled, so when
things freeze we know it is something else.

Best wishes

uriel

On 9/3/07, Charles Forsyth <forsyth@terzarima.net> wrote:
> > The current swap just frustrates people who expect it to work, and
> > then have their systems freeze randomly. Maybe by disabling/remove
> > swap support, then if someone really needs swap he will fix it first
> > and then we can add it back.
>
> i'm not sure all the random freezes are caused by swap.
> in a way, it would be nice if they were, but i wonder...
>
>

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> The current swap just frustrates people who expect it to work, and
> then have their systems freeze randomly. Maybe by disabling/remove
> swap support, then if someone really needs swap he will fix it first
> and then we can add it back.

i'm not sure all the random freezes are caused by swap.
in a way, it would be nice if they were, but i wonder...

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

>> The current swap just frustrates people who expect it to work, and
>> then have their systems freeze randomly. Maybe by disabling/remove
>> swap support, then if someone really needs swap he will fix it first
>> and then we can add it back.
>
> i'm not sure all the random freezes are caused by swap.
> in a way, it would be nice if they were, but i wonder...

probablly not.

i am currently tracking down a problem on my amd system
at home.  the fault seems paging related, but i don't have any
configured "swap" partition.

this fault is during demand loading.

- erik

cpu0: registers for smtpd 36123
FLAGS=10206 TRAP=E ECODE=0 PC=F018BF4C SS=5C88 USP=F025194C
  AX F0059E1C  BX F0059018  CX 00827424  DX 0000F000
  SI 0000F000  DI 00010000  BP 00000000
  CS 0010  DS 0008  ES 0008  FS 001B  GS 001B
  CR0 80010039 CR2 00827458 CR3 08e5c000 CR4 000000d0
  MCA 177e882776 MCT 00000000
  ur f367e4c0 up f02d1ab8
panic: fault: 0x827458

ktrace ->

src(0xf0108b6e); // dumpstack+0x10
src(0xf014ddb6); // panic+0xfe
src(0xf0108d8c); // fault386+0x17d
src(0xf010832f); // trap+0x15d
src(0xf010062a); // forkret
//passing interrupt frame; last pc found at sp=0xf367e4b8
src(0xf018bf4c); // lookpage+0x2f
src(0xf018ab7e); // pio+0x34
src(0xf018aaaf); // fixfault+0x337
src(0xf018a663); // fault+0xc8
src(0xf0108d52); // fault386+0x143
src(0xf010832f); // trap+0x15d
src(0xf010062a); // forkret


acid: src(0xf018bf4c); // lookpage+0x2f
/sys/src/9/port/page.c:441
 4 439		lock(&palloc.hashlock);
 440		for(f = pghash(daddr); f; f = f->hash) {
>441			if(f->image == i && f->daddr == daddr) {

/sys/src/9/port/fault.c:198
  196		if(loadrec == 0) {	/* from a text/data image */
 197			daddr = s->fstart+soff;
>198			new = lookpage(s->image, daddr);
 199			if(new != nil) {

/sys/src/9/port/fault.c:90
  88		case SG_TEXT: 			/* Demand load */
 89			if(pagedout(*pg))
>90				pio(s, addr, soff, pg);

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> One might allocate at least 3.2GB of swap for a 4GB machine, but many
> of our machines run with no swap, and we're probably not alone.  And
> 200 processes are not a lot.  Would you really have over 32GB of swap
> allocated for a 4GB machine with 2,000 processes?
>
> Programs can use a surprising amount of stack space.  A recent notable
> example is venti/copy when copying from a nightly fossil dump score.
> I think we want to be generous about maximum stack sizes.
>
> I don't think that estimates of VM usage would be an improvement.  If
> we can't get it exactly right, there will always be complaints.

if venti/copy's current behavior could be worked around by allocating stuff
instead of using the stack.  we don't have to base design around what
venti/copy does today.

why would it be unacceptable to have a maximum stack allocation
system-wide?  say 16MB.  this would allow is not to overcommit memory.

if we allow overcomitted memory, *any* access of brk'd memory might page
fault.  this seems like a real step backwards in error recovery as most programs
assume that malloc either returns n bytes of valid memory or fails.  since
this assumption is false, either we need to make it true or fix most programs.

upas/fs fails in this way for us all the time.

this would have more serious consequences if, say, venti or fossil suffered
a similar fate.

- erik

Re: [9fans] plan 9 overcommits memory?

[geoff]

If you don't want to swap, rename or delete any swap partitions you
have and don't run swap in termrc nor cpurc.

Re: [9fans] plan 9 overcommits memory?

[geoff]

venti/copy is just an example; programs may legitimately have large
stacks.

If your machines are regularly running out of VM, something is wrong
in your environment.  I would argue that we'd be better off fixing
upas/fs to be less greedy with memory than contorting the system to
try to avoid overcommitting memory.  If one did change the system to
enforce a limit of 16MB for the aggregate of all system stacks, what
would happen when a process needed to grow its stack and the 16MB were
full?  Checking malloc returns cannot suffice.

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> if we allow overcomitted memory, *any* access of brk'd memory might page
> fault.  this seems like a real step backwards in error recovery as most programs
> assume that malloc either returns n bytes of valid memory or fails.  since
> this assumption is false, either we need to make it true or fix most programs.

in the Inferno environment, i concluded that exceptions were the only way of handling
that (you could use notes in Plan 9), and that deals with both explicit
and implicit allocations.  it's more obvious in Inferno because implicit allocations
are common, because the run time system allocates memory dynamically, and not
just for stack frames.

the exception handlers are placed, optionally, fairly high up in the application processes, with further supervising sets
towards the root of the system (eg, to encapsulate individual applications within the window system).
an unhandled exception within an application process causes that process and perhaps others in its group
to die, and the exception is propagated to a process that's the nominated process group leader.

note that the process that incurs the exception is just the one that ran out of memory, not the one
that `really' caused the problem.  there needs to be some extra mechanism to ensure that important
functions survive in any case.

i looked at quota systems, but they are far too pessimistic for memory systems as they are for disc systems
(for most embedded devices, which is where you typically care most about this).
typically you end up with either over-committing (which is where we started), or poor utilisation
(which also isn't great for small embedded systems).

that left some form of allocation accounting, but we found that most programmers
for one reason or another found quite hard the systems analysis that's needed to make
allocation accounting work (although the degree of pessimism is typically much less
than that of quotas, which are too coarse-grained).

i used a variant that reserved a given amount of memory for use by distinguished processes
critical to system operation or recovery.   (perhaps this protected memory structure should have nested,
but it seemed better to see if that would really be useful.)

systems analysis at this level is much easier, though still neglected.

Re: [9fans] plan 9 overcommits memory?

[ron minnich]

One option for Erik: try changing the segment allocator so that it
faults in all segment pages on creation. Would this do what you want?
I will try this if I get time later today. Assuming it is as simple as
my simple-minded description makes it sound.

If it would, maybe a simple
echo faultall > /proc/pid/ctl
would be useful

would be interesting: iterate over all segments, and make sure each
has a real page for all pages in all segments.

I can see the need for not overcommitting, and also for actually
creating and filling out the pages on malloc or other allocation.
Indeed, lack of OS thrashing due to paging is one feature cited by
proponents of this:
http://www.cs.sandia.gov/~smkelly/SAND2006-2561C-CUG2006-CatamountDualCore.pdf

ron

Re: [9fans] plan 9 overcommits memory?

[Eric Van Hensbergen]

I was thinking that this was probably what we wanted to do for HPC....
also, having the option of turning off zero-filling pages....

     -eric


On 9/3/07, ron minnich <rminnich@gmail.com> wrote:
> One option for Erik: try changing the segment allocator so that it
> faults in all segment pages on creation. Would this do what you want?
> I will try this if I get time later today. Assuming it is as simple as
> my simple-minded description makes it sound.
>
> If it would, maybe a simple
> echo faultall > /proc/pid/ctl
> would be useful
>
> would be interesting: iterate over all segments, and make sure each
> has a real page for all pages in all segments.
>
> I can see the need for not overcommitting, and also for actually
> creating and filling out the pages on malloc or other allocation.
> Indeed, lack of OS thrashing due to paging is one feature cited by
> proponents of this:
> http://www.cs.sandia.gov/~smkelly/SAND2006-2561C-CUG2006-CatamountDualCore.pdf
>
> ron
>

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> One option for Erik: try changing the segment allocator so that it
> faults in all segment pages on creation. Would this do what you want?
> I will try this if I get time later today. Assuming it is as simple as
> my simple-minded description makes it sound.

grudgingly, i admit it would -- assuming that malloc then returns 0 and
doesn't send the proc a note.

you've got me motivated to think about this some more.

- erik

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

On Mon Sep  3 16:48:59 EDT 2007, geoff@plan9.bell-labs.com wrote:
> If your machines are regularly running out of VM, something is wrong
> in your environment.  I would argue that we'd be better off fixing
> upas/fs to be less greedy with memory than contorting the system to
> try to avoid overcommitting memory.

well, yes.  the problem is 400MB mailboxes.  but i'll let you tell
folk with mailboxes that large, that that's too large. ;-)

it'd be nice to be able to use more than 3.75-pcispace GB of memory.

but i don't see this as a "fix upasfs" problem.  i see this as a general
problem that upas/fs's huge memory usage highlights.  this can happen
to any process.  suppose i start a program that allocates 8k but between
the malloc and the memset, another program uses the last available
page in memory, then my original program faults.

> If one did change the system to
> enforce a limit of 16MB for the aggregate of all system stacks, what
> would happen when a process needed to grow its stack and the 16MB were
> full?  Checking malloc returns cannot suffice.

no, it wouldn't.  obviously one doesn't malloc the stack -- at least not today.
but this is no worse than the current situation for stacks.
and an improvement for the heap.

if one made the limit settable at runtime, one could verify reasonable
stack usage while testing.

here i think ron's idea of pre-faulting makes even more sense for
the stack than the heap, as stack allocation is implicit.

- erik

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> to any process.  suppose i start a program that allocates 8k but between
> the malloc and the memset, another program uses the last available
> page in memory, then my original program faults.

yes, and you'll always have to deal with that in some form or another.
i've started a program, it allocates some memory, is guaranteed to have it (unlike the current system),
but later, some other program allocates enough memory that mine can't get
any more, memory that mine needs to finish (perhaps during an output phase).
my original program fails, even though the system guarantees physical memory
for all virtual memory allocations.

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

>> to any process.  suppose i start a program that allocates 8k but between
>> the malloc and the memset, another program uses the last available
>> page in memory, then my original program faults.
>
> yes, and you'll always have to deal with that in some form or another.
> i've started a program, it allocates some memory, is guaranteed to have it (unlike the current system),
> but later, some other program allocates enough memory that mine can't get
> any more, memory that mine needs to finish (perhaps during an output phase).
> my original program fails, even though the system guarantees physical memory
> for all virtual memory allocations.

that would be perfect.

perhaps i've been unclear.  i don't have any problem dealing with failed
alloc.  malloc has always been able to return 0.

dealing with a page fault due to overcommit is a different story.

- erik

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> perhaps i've been unclear.  i don't have any problem dealing with failed
> alloc.  malloc has always been able to return 0.
>
> dealing with a page fault due to overcommit is a different story.

that's a slightly different aspect.  the note should not be "page fault" but
"out of memory" (or some such thing).  that's much better than a nil return.
most errors on shared resoruces are better expressed as exceptions (notes),
because that's what they are: they are a failure of the underlying physical or virtual machine
to handle an exceptional case.  the code shouldn't have to deal with it explicitly everywhere,
except in C to detect and propagate the exception to code that knows what's going on.

exceptions have acquired a bad name in some circles because of the way that some
people tried to use them for situations, usually in interfaces, that are hardly exceptional (eg, Ada and Java).

Re: [9fans] plan 9 overcommits memory?

[lucio]

> No, and it would be hard to do it because you'd need ways to compact
> fragmented memory after a lot of mallocs and frees.  And then, you'd
> need a way to fix the pointers after compacting.

Is it all localised, or is the code scattered across multiple kernel
modules?  Many years ago I put a lot of effort into a scheme for
automatic compacting of memory that worked pretty efficiently on an
8088 without being particularly architecture dependent.  When I tried
to implement it on NetBSD, it turned out that I did not understand
NetBSD particularly well and that was the last of my attempts.

The code has long been lost, but I remember the strategy very well, so
I could try again.

++L

Re: [9fans] plan 9 overcommits memory?

[Dave Eckhardt]

> On the pc, Plan 9 currently limits user-mode stacks to 16MB.
> On a CPU server with 200 processes (fairly typical), that's
> 3.2GB of VM one would have to commit just for stacks.  With
> 2,000 processes, that would rise to 32GB just for stacks.

There's probably no simple answer which is correct for all goal
sets.

For an embedded widget, you might want to create a small number
of processes and be utterly sure none of them would run out of
RAM (which might be small).  If you think your stuff fits in
small stacks you'd probably like to know as early as possible
if it doesn't, so the kernel "helpfully" giving you 16-meg
stacks might not be so helpful.

For a web server you probably want some number of parallel
requests to run to completion and excess requests to be queued
and/or retried by the (remote) browser.  Overcommitting seems
likely to be harmful here, since each process which dies when
it tries to grow a stack page won't complete, and may return
a nonsense answer to the client.  It seems like you could thrash,
with most processes running for a while before getting killed.

Overcommitted 16-meg stacks are probably fine for lightly-loaded
CPU servers running random mixes of programs... but I suspect
other policies would also be fine for this case.

Personally I am not a fan of programs dying randomly because of
the behavior of other programs.  So I guess my vote would be for
a small committed stack (a handful of pages) with an easy way for
processes with special needs to request a (committed) larger size.

But I'd probably prefer an OHCI USB driver first :-)

Dave Eckhardt

Re: [9fans] plan 9 overcommits memory?

[Douglas A. Gwyn]

<geoff@plan9.bell-labs.com> wrote in message
news:01b719eaabe004a9073ccb4b3425e1d0@plan9.bell-labs.com...
> ... The big exception is stack growth.  ...

That has indeed been a longstanding problem, and if the OSes
want to grow up they need to solve that problem.  It is obvious
how to solve it if speed isn't an issue; just test upon each function
invocation.  I bet there are efficient hacks for that..

Re: [9fans] plan 9 overcommits memory?

[Douglas A. Gwyn]

<john@csplan9.rit.edu> wrote...
> there's no conceivable reason anyone would want swap,  and operating
> systems with working swap suck ;)

Actually there have been many successful OSes with swapping/demand paging.

A way to make it work is for process initiation to include resource
allocation, especially memory limits.  Or, you could implement the
"working set" concept according to which a process swaps only
against itself., and the maximum working set RAM is guaranteed or
the attempt to execute the process reports failure at the outset.

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> that's a slightly different aspect.  the note should not be "page fault" but
> "out of memory" (or some such thing).  that's much better than a nil return.
> most errors on shared resoruces are better expressed as exceptions (notes),
> because that's what they are: they are a failure of the underlying physical or virtual machine
> to handle an exceptional case.  the code shouldn't have to deal with it explicitly everywhere,
> except in C to detect and propagate the exception to code that knows what's going on.

if one wishes to be remotely standards-compliant, sending a note on allocation
failure is not an option.  k&r 2nd ed. p. 252.

- erik

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> if one wishes to be remotely standards-compliant, sending a note on allocation
> failure is not an option.  k&r 2nd ed. p. 252.

i was discussing  something about it in practice, and not in a 1970's environment,
where the approach didn't really work well even then.  the `recovery' that resulted was almost
invariably equivalent to sysfatal.

Re: [9fans] plan 9 overcommits memory?

[ron minnich]

Two cases so far: running out of stack and allocating overcommited memory.

You can easily catch stack growth failure in the OS. The proc gets a
note. The proc has the
option of the equivalent of 'echo growstack xyz > /proc/me/ctl'.

For overcommits, 'echo faultall > /proc/me/ctl'.

can we catch everything this way?

ron

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

i can't remember whether anyone pointed this out as well (probably):
you'll need to ensure that each fork reserves as many physical pages as are currently
shared in the data space, for the life of the shared data,
so that every subsequent copy-on-write is guaranteed to succeed.
this will prevent some large processes from forking to exec much smaller images.

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> > if one wishes to be remotely standards-compliant, sending a note on allocation
> > failure is not an option.  k&r 2nd ed. p. 252.
>
> i was discussing  something about it in practice, and not in a 1970's environment,
> where the approach didn't really work well even then.  the `recovery' that resulted was almost
> invariably equivalent to sysfatal.

sysfatal is a reasonable recovery strategy for many programs.  for many others,
there may be something useful to do, like allocate a smaller or fewer buffers.

- erik

Re: [9fans] plan 9 overcommits memory?

[sqweek]

On 9/4/07, erik quanstrom <quanstro@quanstro.net> wrote:
> > Also, [swap is] broken, broken, broken on Plan 9
>
> but could you describe what antisocial behavior it exhibits and how one
> could reproduce this behavior?

 My cpu/auth/file server is a poor little headless P100 with 24MB RAM
(there's 32 in there but apparently one of the sticks is faulty). I
have a 192MB swap partition set up, man -P hoses the box (gs was the
biggest memory user IIRC). Actually, hoses is a bit misleading...
 I hear the box reading the disk for a time, then my drawterm locks
up, then it carries on with the disk activity, changes sound slightly
(guess it's into swap), and finally goes silent. drawterm is still
locked up, it's like something swap related is deadlocked or somesuch.
 Now, I'm sure, in the past that if I ^T^Tr drawterm at this point,
some time later the box recovers in a flourish of disk activity and I
can reconnect. But apparently this is not guarenteed as last night
when I tried it to get accurate timings it really was hosed, and still
dead when I woke up just now. I dragged a monitor over to it but my ps
hung, so I guess fossil or something else important bit it.
Unfortunately I forgot about ^T^Tp until just now.

 So yeah, I've probably got a decent test bed for swapping.
-sqweek

Re: [9fans] plan 9 overcommits memory?

[Douglas A. Gwyn]

Charles Forsyth wrote:
> you'll need to ensure that each fork reserves as many physical pages as are currently
> shared in the data space, for the life of the shared data,
> so that every subsequent copy-on-write is guaranteed to succeed.
> this will prevent some large processes from forking to exec much smaller images.

That's why many OSes have a "spawn" primitive that combines fork-and-exec.

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> That's why many OSes have a "spawn" primitive that combines fork-and-exec.

the problem with spawn is it requires a mechanism to replace that little
block of code between the fork and exec.  that code is hardly ever the
same so spawn keeps growing arguments.

- erik

Re: [9fans] plan 9 overcommits memory?

[Douglas A. Gwyn]

erik quanstrom wrote:
> > That's why many OSes have a "spawn" primitive that combines fork-and-exec.
> the problem with spawn is it requires a mechanism to replace that little
> block of code between the fork and exec.  that code is hardly ever the
> same so spawn keeps growing arguments.

Yes, on the other hand I bet a spawn interface could be devised that
is sufficiently programmable.  (May require some extra dup()s etc. to
really handle all the common usage patterns.)

Re: [9fans] plan 9 overcommits memory?

[Joel C. Salomon]

> > > That's why many OSes have a "spawn" primitive that combines fork-and-exec.
> >
> > the problem with spawn is it requires a mechanism to replace that little
> > block of code between the fork and exec.  that code is hardly ever the
> > same so spawn keeps growing arguments.
>
> Yes, on the other hand I bet a spawn interface could be devised that
> is sufficiently programmable.  (May require some extra dup()s etc. to
> really handle all the common usage patterns.)

What could accomplish this are hooks into the process-creation bits of
the kernel.  I've lost the links to this, but some toy OS accomplished
process creation with the equivalent of 'cp /bin/ls /proc/clone'.
This isn't useful on its own, but it could be an interesting exercise
to construct a process outside the kernel and kick-start it somehow.

The one somewhat useful application of this idea would be to mitigate
the bug documented in notify(2):
    Since exec(2) discards the notification handler, there
    is a window of vulnerability to notes in a new process.
In a hand-rolled spawn(), one could conceivably install a notification
handler before the process starts executing or accepting notes.

Possibly some form of dynld() could be built on this too.

I can't imagine that either of these uses are nearly compelling enough
to open this can of worms....  Has anyone truly felt confined by Plan
9's fork+exec model?

--Joel

Re: [9fans] plan 9 overcommits memory?

[ron minnich]

On 9/6/07, Joel C. Salomon <joelcsalomon@gmail.com> wrote:

> I can't imagine that either of these uses are nearly compelling enough
> to open this can of worms....  Has anyone truly felt confined by Plan
> 9's fork+exec model?

yes, because exec takes a pathname. that's a pull model. That is
pretty awful in a large machine. Define awful: ok, it's the difference
between startup times of 3+ minutes, 400 nodes, vs. 3 seconds. That's
awful.

it's why we started doing xcpu in the first place: push the binary to
a ram disk, then at least xcpu is pulling from a local place, not a
network. But xcpu was a compromise: I really wanted to do a process
creation device.

ron

Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth]

> yes, because exec takes a pathname. that's a pull model. That is

9p can remove your pain

Re: [9fans] plan 9 overcommits memory?

[Steve Simon]

I wonder if this is interesting but the user mode processes creation device is,
in effect, what I and brucee have been working on (seperately) for plan9 binaries
on windows and what russ did for linux binaries on plan9.

BTW the latter has seen some development recently - somone managed to get dynamic libs
working with russ's code (no major work, more a hassle I beleive) and got as far as
getting dillo to run. The downside is dillo has to render its output on an X11 display...

still, makes you think...

-Steve

Re: [9fans] plan 9 overcommits memory?

[Roman Shaposhnik]

On Thu, 2007-09-06 at 12:38 -0700, ron minnich wrote:
> On 9/6/07, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
>
> > I can't imagine that either of these uses are nearly compelling enough
> > to open this can of worms....  Has anyone truly felt confined by Plan
> > 9's fork+exec model?
>
> yes, because exec takes a pathname. that's a pull model. That is
> pretty awful in a large machine. Define awful: ok, it's the difference
> between startup times of 3+ minutes, 400 nodes, vs. 3 seconds. That's
> awful.
>
> it's why we started doing xcpu in the first place: push the binary to
> a ram disk, then at least xcpu is pulling from a local place, not a
> network. But xcpu was a compromise: I really wanted to do a process
> creation device.

  Exposing an interface for process manipulation to userspace would be
quite cool. Any prototypes on Plan9 so far?

Thanks,
Roman.

Re: [9fans] plan 9 overcommits memory?

[Bruce Ellis]

this thread is now twice as big as the swap and proc code.
is it really that much fun to do research this way?
yes, it has been done.  no, my current work has nothing
to do with this issue.

someone written a line of relevant code during this "discussion"?

brucee

On 9/7/07, Roman Shaposhnik <rvs@sun.com> wrote:
> On Thu, 2007-09-06 at 12:38 -0700, ron minnich wrote:
> > On 9/6/07, Joel C. Salomon <joelcsalomon@gmail.com> wrote:
> >
> > > I can't imagine that either of these uses are nearly compelling enough
> > > to open this can of worms....  Has anyone truly felt confined by Plan
> > > 9's fork+exec model?
> >
> > yes, because exec takes a pathname. that's a pull model. That is
> > pretty awful in a large machine. Define awful: ok, it's the difference
> > between startup times of 3+ minutes, 400 nodes, vs. 3 seconds. That's
> > awful.
> >
> > it's why we started doing xcpu in the first place: push the binary to
> > a ram disk, then at least xcpu is pulling from a local place, not a
> > network. But xcpu was a compromise: I really wanted to do a process
> > creation device.
>
>  Exposing an interface for process manipulation to userspace would be
> quite cool. Any prototypes on Plan9 so far?
>
> Thanks,
> Roman.
>
>

Re: [9fans] plan 9 overcommits memory?

[Lyndon Nerenberg]

On 2007-Sep-6, at 21:09 , Bruce Ellis wrote:

> someone written a line of relevant code during this "discussion"?

Be careful.  Coding to mailing list discussions results in Linux.

FWIW, the discussion here has made more sense than any and all
arguments/conversations I've had with UNIX vendors over the last
decade-and-a-half.

And I'm talking about both malloc and spawn.

The concept of assembling processes in user space is intriguing, and
I'd like to hear more from the people who have thought/played with
the idea.

I'm convinced there is no solution to brk; it's an untenable
implementation (although it made perfect sense at the time).  As long
as our programming languages insist on direct memory pointers I don't
see a way out.  But so far all the alternatives just lead to madness ...

--lyndon

Re: [9fans] plan 9 overcommits memory?

[Bruce Ellis]

coding at home and getting something significant working
is what i meant.  but by all means write to the list instead.

brucee

On 9/7/07, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
>
> On 2007-Sep-6, at 21:09 , Bruce Ellis wrote:
>
> > someone written a line of relevant code during this "discussion"?
>
> Be careful.  Coding to mailing list discussions results in Linux.
>
> FWIW, the discussion here has made more sense than any and all
> arguments/conversations I've had with UNIX vendors over the last
> decade-and-a-half.
>
> And I'm talking about both malloc and spawn.
>
> The concept of assembling processes in user space is intriguing, and
> I'd like to hear more from the people who have thought/played with
> the idea.
>
> I'm convinced there is no solution to brk; it's an untenable
> implementation (although it made perfect sense at the time).  As long
> as our programming languages insist on direct memory pointers I don't
> see a way out.  But so far all the alternatives just lead to madness ...
>
> --lyndon
>

Re: [9fans] plan 9 overcommits memory?

[Lyndon Nerenberg]

On 2007-Sep-6, at 21:37 , Bruce Ellis wrote:

> coding at home and getting something significant working
> is what i meant.  but by all means write to the list instead.

So how do you see the way out of a brk() implementation while still
coding in a pointer based environment?  Code only follows engineering.

If I even Thought About Mentioning GC, well, ... let me put up the
window shutters.

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> this thread is now twice as big as the swap and proc code.
> is it really that much fun to do research this way?
> yes, it has been done.  no, my current work has nothing
> to do with this issue.
>
> someone written a line of relevant code during this "discussion"?

would you care to give references to this being done
in plan 9 or similar operating system?

i would hope the discussion is much bigger than the
resulting code.  this discussion has boiled down the
issues for me.  this is what i remember

plan 9 overcommits memory three ways
1. when a process forks, the cow pages are counted once.
2. plan 9 hands out brk'ed pages it doesn't have.
3. the stack is potentially very large (16mb), grown
implictly and not reserved.

by the way, what are the scare quote quotes for?
participants might take that in the wrong way.

- erik

Re: [9fans] plan 9 overcommits memory?

[roger peppe]

brucee wrote:
> coding at home and getting something significant working
> is what i meant.  but by all means write to the list instead.

i agree totally that it's a Good Thing to go and actually code something
up, but i also think that it's a positive thing to have technical discussions
out in the open. not necessarily because it's an efficient way to do things
(it probably isn't), but because it keeps a useful record that people
can later use to learn from.

i heard someone on another list say it nice and succinctly:

: FWIW, I strongly prefer to have as much of technical discussions on-list
: as possible: that way they benefit future generations of SBCL hackers as
: well.

Re: [9fans] plan 9 overcommits memory?

[sqweek]

On 9/5/07, ron minnich <rminnich@gmail.com> wrote:
> On 9/4/07, sqweek <sqweek@gmail.com> wrote:
> >  Well, I guess the next question is: Is malloc's interface sufficient/practical?
>
> sure. I use it all the time, I'm using it now.
>
> I use it several hundred times a second without knowing it, I bet.

 How many times does it return NULL?

Re: [9fans] plan 9 overcommits memory?

[ron minnich]

On 9/4/07, sqweek <sqweek@gmail.com> wrote:

>  Well, I guess the next question is: Is malloc's interface sufficient/practical?

sure. I use it all the time, I'm using it now.

I use it several hundred times a second without knowing it, I bet.

ron

Re: [9fans] plan 9 overcommits memory?

[sqweek]

On 9/3/07, erik quanstrom <quanstro@quanstro.net> wrote:
>         if((p = malloc(Size)) == 0)
>                 /* malloc recovery code */
>                 /* why bother?  the kernel could be lying to us anyway. */

 Having run into issues writing codes with large memory footprints on
linux (which also overcommits [by default]), erik's comment here is
close to my heart. Indeed, what's the point of a malloc interface that
returns 0 when in practice the kernel kills your process before that
happens? It strikes me as irresponsible to advertise an interface you
won't/can't honour.
 This is certainly my biggest problem with overcommitting. Well, that
and the non determinism, but it's not clear to me that's avoidable
when memory is exhausted.

geoff@plan9.bell-labs.com wrote:
> The big exception is stack growth.  The
> kernel automatically extends a process's stack segment as needed.

 OK, so the stack takes precedence over heap allocations. What if it
was the other way around? Instead of stealing pages that process Y has
malloc'd but not yet touched to grow the stack of process X, kill
process X because there's not enough memory left to grow the stack...
 Mmmm. I don't think I helped anything. On the surface it looks like
I'm honouring the malloc request, but there's probably a fair chance X
== Y and the non-determinism really is the problem.

 Well, I guess the next question is: Is malloc's interface sufficient/practical?
-sqweek

Re: [9fans] plan 9 overcommits memory?

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On 9/4/07, erik quanstrom <quanstro@coraid.com> wrote:
>
> On Tue Sep  4 09:39:37 EDT 2007, leimy2k@gmail.com wrote:
>
> > Yep, I've seen code with totally erroneous use of realloc work perfectly
> on
> > Linux for example, due to it's behavior.  Then I built it on FreeBSD and
> it
> > failed appropriately :-).
>
> what does this have to do with memory overcommitment?  k&r second ed,
> p. 252 specifies that malloc and realloc return NULL if the request can't
> be specified.
>
> - erik
>

Plenty if I shared the code with you... :-)  Has to do with what memory was
allocated vs what was being used.

Unfortunately I'm not at liberty to post such code.


Let's just say it shouldn't have returned NULL in this case (to my
recollection).  Different OSes behave differently with what is committed via
the allocation functions of stdlib.  I'm thinking they're creeping into
undefined behavior territory of the standard rather than modes that are
explicitly supposed to fail.

Dave

[-- Attachment #2: Type: text/html, Size: 1544 bytes --]

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

On Tue Sep  4 09:39:37 EDT 2007, leimy2k@gmail.com wrote:

> Yep, I've seen code with totally erroneous use of realloc work perfectly on
> Linux for example, due to it's behavior.  Then I built it on FreeBSD and it
> failed appropriately :-).

what does this have to do with memory overcommitment?  k&r second ed,
p. 252 specifies that malloc and realloc return NULL if the request can't
be specified.

- erik

Re: [9fans] plan 9 overcommits memory?

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1830 bytes --]

On 9/4/07, Douglas A. Gwyn <DAGwyn@null.net> wrote:
>
> >> malloc just moves the brk, but the backing pages don't get
> >> allocated until the pages are accessed (during memset).
>
> "erik quanstrom" <quanstro@quanstro.net> wrote in message
> news:d7c705e9c3355e723c8f69677bf2d851@quanstro.net...
>
> > i'm just suprised that plan 9 overcommits.  this makes
> > this code nonsensical from user space
> > if((p = malloc(Size)) == 0)
>
> Indeed, when I discovered that Linux was overcommitting memory
> in much the same way, which in my view is erroneous design, I
> added an optional feature to my portable malloc implementation
> that (on systems neeing it) would touch every allocated page
> before reporting success.  (During the touch loop a trap catcher
> would set a flag that would be tested after the loop to determine
> whether there had ben any faults.)  malloc really shouldn't have
> to do this, but if you're insisting on overcommitment in the OS
> then the library malloc() needs to do it.  Otherwise, applications
> that try to be careful may still fail "randomly", which is intolerable.
>

Yep, I've seen code with totally erroneous use of realloc work perfectly on
Linux for example, due to it's behavior.  Then I built it on FreeBSD and it
failed appropriately :-).

The problem there, in my opinion, was not the OS's behavior, but the
inexperience of the coder in question with realloc.  The terrible side
effect of all these garbage collected languages or ones that don't make you
manage memory much is that programmers have this attitude that they aren't
to blame when their code has flaws.

Mac OS X has many options for testing malloc results as well.  It can
scribble on pages etc... I'm sure glibc has these options somewhere too.
 They're great testing tools.

Dave

[-- Attachment #2: Type: text/html, Size: 2552 bytes --]

Re: [9fans] plan 9 overcommits memory?

[Douglas A. Gwyn]

"erik quanstrom" <quanstro@quanstro.net> wrote in message
news:d81b7ef468d08ecf9e31985f92a9b0c4@quanstro.net...
> but why introduce unpredictability?  who really programs as if
> memory is not overcommited?

Practically everybody.

> i would bet that acme and most
> residents of /sys/src/cmd could do quite bad things to you in these
> cases.  there's no waserror() in userland to wrap around memset.

You don't have to do that if malloc takes are of it.  (See my nearby
posting.)

> how much memory can be wasted by assuming that all brk'ed memory
> can be used?

If you allocate the storage, presumably you should expect to use it.

Anyway, it's better to be wasteful and correct than randomly failing.

Re: [9fans] plan 9 overcommits memory?

[Douglas A. Gwyn]

>> malloc just moves the brk, but the backing pages don't get
>> allocated until the pages are accessed (during memset).

"erik quanstrom" <quanstro@quanstro.net> wrote in message
news:d7c705e9c3355e723c8f69677bf2d851@quanstro.net...
> i'm just suprised that plan 9 overcommits.  this makes
> this code nonsensical from user space
> if((p = malloc(Size)) == 0)

Indeed, when I discovered that Linux was overcommitting memory
in much the same way, which in my view is erroneous design, I
added an optional feature to my portable malloc implementation
that (on systems neeing it) would touch every allocated page
before reporting success.  (During the touch loop a trap catcher
would set a flag that would be tested after the loop to determine
whether there had ben any faults.)  malloc really shouldn't have
to do this, but if you're insisting on overcommitment in the OS
then the library malloc() needs to do it.  Otherwise, applications
that try to be careful may still fail "randomly", which is intolerable.

Re: [9fans] plan 9 overcommits memory?

[Scott Schwartz]

On Sun, Sep 02, 2007 at 06:47:17PM -0700, ron minnich wrote:
> If you can't live with overcommit, maybe you need a wrapper that:
> sets up to catch the note (I am assuming here that you get one; do you?)

That's still a race.  Getting all the memory at once is different from
probing for one page at a time and dealing with failures in the middle.

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> but most people can live with the overcommit, as witness the fact that
> most of us do and never know it.
>
> If you can't live with overcommit, maybe you need a wrapper that:
> sets up to catch the note (I am assuming here that you get one; do you?)
> malloc
> zero memory you malloc'ed (which will get the pages in)
> die reasonably if you get the note

but why introduce unpredictability?  who really programs as if
memory is not overcommited?  i would bet that acme and most
residents of /sys/src/cmd could do quite bad things to you in these
cases.  there's no waserror() in userland to wrap around memset.

how much memory can be wasted by assuming that all brk'ed memory
can be used?

- erik

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> but most people can live with the overcommit, as witness the fact that
> most of us do and never know it.
>
> If you can't live with overcommit, maybe you need a wrapper that:
> sets up to catch the note (I am assuming here that you get one; do you?)
> malloc
> zero memory you malloc'ed (which will get the pages in)
> die reasonably if you get the note

but why introduce unpredictability?  who really programs as if
memory is not overcommited?  i would bet that acme and most
residents of /sys/src/cmd could do quite bad things to you in these
cases.  there's no waserror() in userland to wrap around memset.

how much memory can be wasted by assuming that all brk'ed memory
can be used?

- erik

Re: [9fans] plan 9 overcommits memory?

[ron minnich]

but most people can live with the overcommit, as witness the fact that
most of us do and never know it.

If you can't live with overcommit, maybe you need a wrapper that:
sets up to catch the note (I am assuming here that you get one; do you?)
malloc
zero memory you malloc'ed (which will get the pages in)
die reasonably if you get the note

Would that work?

ron

Re: [9fans] plan 9 overcommits memory?

[Scott Schwartz]

Russ:
| you could argue for some kind of accounting that would
| ensure pages were available, but this could only be
| terribly pessimistic, especially in the case of stacks
| and fork.

Still, that's the way unix worked.  You can deal with the pessimism by
allocating lots of backing store, whereas with overcommit (at least on
linux) you just have to learn to live with processes dying randomly.

But you knew all that. :)

Re: [9fans] plan 9 overcommits memory?

[erik quanstrom]

> the problem is not really as easy as it might seem at first.
> malloc just moves the brk, but the backing pages don't get
> allocated until the pages are accessed (during memset).
>

i'm just suprised that plan 9 overcommits.  this makes
this code nonsensical from user space

	if((p = malloc(Size)) == 0)
		/* malloc recovery code */
		/* why bother?  the kernel could be lying to us anyway. */

also, by default plan 9 assumes that there are 16 MB of memory
more than actually exist.  i would think with today's large memories
typically, one would want to assume the worst and underestimate
the amount of memory required so malloc would fail rather than
getting a page fault later.

> you could argue for some kind of accounting that would
> ensure pages were available, but this could only be
> terribly pessimistic, especially in the case of stacks
> and fork.

how pessimistic would this be?  i'll gladly trade a substantial
chunk of memory for failure on allocation rather than failure
on access.

- erik

Re: [9fans] plan 9 overcommits memory?

[Russ Cox]

> this means that the malloc *succeeded* it wasn't until i forced
> the pagefault with the memset that i ran out of memory.  what's
> going on here?

you know what's going on here.  read the subject you wrote.

the problem is not really as easy as it might seem at first.
malloc just moves the brk, but the backing pages don't get
allocated until the pages are accessed (during memset).

you could argue for some kind of accounting that would
ensure pages were available, but this could only be
terribly pessimistic, especially in the case of stacks
and fork.

russ

[9fans] plan 9 overcommits memory?

[erik quanstrom]

i was trying to tickle a kernel panic, but instead
i think i found a bug.  this program was run on
a machine with 1800 MB user space available.
(3552/464510 user)

	#include<u.h>
	#include<libc.h>
	
	enum{
		Big	= 1024*1024*1790,
	};
	
	void
	main(void)
	{
		char *p;
		ulong i;
	
		p = malloc(Big);
		if(p == 0)
			sysfatal("malloc: %r");
		memset(p, 0, Big);
		exits("");
	}

the result was

	ladd# out of physical memory; no swap configured
	364: 8.out killed: out of memory

this means that the malloc *succeeded* it wasn't until i forced
the pagefault with the memset that i ran out of memory.  what's
going on here?

- erik