Re: malloc & uncommitting memory

[Rich Felker <dalias@aerifal.cx>]

On Thu, Apr 03, 2014 at 08:17:25PM +0200, M. Ziebell wrote:
> As far as i understand "uncommitted" memory is allocated address space.
> A region in memory which is not used by the program, but the address
> space expanded to make a smaller region available to the programm for
> use.
> Please correct me if I am wrong.

I'm not clear on exactly what you're asking so I'll try to just
clarify the issue. There are basically three types of memory usage
that matter:

- Virtual address space: Process-local resource. Aside from relatively
  low overhead in kernel bookkeeping, this only matters in that, if
  you run out of addresses, you can't allocate anything else.

- Anonymous pages: Global resource. These are pages that actually
  consume (rather than just temporarily using as cache) physical
  storage (ram or swap) because they're not backed by a file or device
  or the zero page or shared (via COW) with another mapping.

- Commit charge: Global resource. A superset of anonymous pages, also
  counting mapped pages that are presently shared (COW) with the zero
  page, a file, another process, etc. but which will need their own
  storage if/when they're written.

At present, musl's malloc can deallocate only the following:

* All three, only when resizing or freeing an allocation that was
  large enough to be serviced individually by mmap.

* Anonymous pages only, by calling madvise with MADV_DONTNEED to reset
  them to COW copies of the zero page when a large free range
  coalesces in the heap.

In particular, neither commit charge nor virtual address space from
the heap used for small allocations is ever freed. Avoiding freeing
the virtual address space is generally a good thing, since doing so
could result in pathological fragmentation. But not freeing the commit
charge means that an application which momentarily used a large amount
of memory by allocating a small bit at a time (think hideous tree/list
structures used in C++ and glib-based code) then freeing it all will
continue to tie part of the commit limit and reduce the amount of
memory available to other programs (note: this only applies with
overcommit disabled).

> Based on that understanding, the idea or your proposal sounds not
> really desirable.
> You have to take care of tons of exceptions and special cases.
> I'm pretty sure I don't understand what uncommitted memory is for, but
> I'm heary is hundreds of lines of complex code, if-else constructs
> everywhere and bugs.

The whole of malloc is under 600 lines, so anything measured in
"hundreds of lines" is definitely inappropriate as an addition.
However the basic concept (frozen chunk list, described in the first
mail) is at most tens of lines of code, and the strategies 1-3 for
avoiding fragmentation would each be somewhere on the same order of
magnitude, I think.

Rich