Re: [9fans] plan 9 overcommits memory?

[Charles Forsyth <forsyth@terzarima.net>]

> if we allow overcomitted memory, *any* access of brk'd memory might page
> fault.  this seems like a real step backwards in error recovery as most programs
> assume that malloc either returns n bytes of valid memory or fails.  since
> this assumption is false, either we need to make it true or fix most programs.

in the Inferno environment, i concluded that exceptions were the only way of handling
that (you could use notes in Plan 9), and that deals with both explicit
and implicit allocations.  it's more obvious in Inferno because implicit allocations
are common, because the run time system allocates memory dynamically, and not
just for stack frames.

the exception handlers are placed, optionally, fairly high up in the application processes, with further supervising sets
towards the root of the system (eg, to encapsulate individual applications within the window system).
an unhandled exception within an application process causes that process and perhaps others in its group
to die, and the exception is propagated to a process that's the nominated process group leader.

note that the process that incurs the exception is just the one that ran out of memory, not the one
that `really' caused the problem.  there needs to be some extra mechanism to ensure that important
functions survive in any case.

i looked at quota systems, but they are far too pessimistic for memory systems as they are for disc systems
(for most embedded devices, which is where you typically care most about this).
typically you end up with either over-committing (which is where we started), or poor utilisation
(which also isn't great for small embedded systems).

that left some form of allocation accounting, but we found that most programmers
for one reason or another found quite hard the systems analysis that's needed to make
allocation accounting work (although the degree of pessimism is typically much less
than that of quotas, which are too coarse-grained).

i used a variant that reserved a given amount of memory for use by distinguished processes
critical to system operation or recovery.   (perhaps this protected memory structure should have nested,
but it seemed better to see if that would really be useful.)

systems analysis at this level is much easier, though still neglected.