Re: [Caml-list] Understanding usage by the runtime

[orbitz@ezabel.com]

Thank you for the detailed response Rich.

Isn't the goal of compaction to keep all of these blocks of memory as close as possible?  I should have noted the fragmentation of my heap after compaction, but it seems unlikely that my < 1meg of actual data could be fragmented across 400megs worth of chunks.

> Here's the news: the OS doesn't need you to give back
> memory.  Because of virtual memory and swap, the OS will quite happily
> take back your memory whenever it wants without asking you. 

For most cases this is true, however in my case (which is not the usual case), my OS has no swap.  We actually prefer things to fail than to be swapped because we are doing computations could take months if we get into a swapping situation.  I'm no linux expert so our solution is to not have swap and to keep our VMs light when it comes to I/O.  Perhaps this is a poor solution but it does change things for us.

Thanks again Rich.


On Jan 8, 2012, at 1:45 PM, Richard W.M. Jones wrote:

> On Sat, Jan 07, 2012 at 12:43:22AM -0500, orbitz@ezabel.com wrote:
>> One question does remain though: In my tests I would do some work
>> that would cause ~1GB of RAM to be under control of the Gc.  Then I
>> would do something that, at the time I didn't understand, would case
>> the Gc to compact all of its memory and go back down to less than 1
>> meg, but the RES value in top would only drop to about 400 megs.  Is
>> this expected behavior?  I know the malloc implementation might hold
>> on to some data for itself but 400x the amount of memory the Ocaml RTS
>> actually needs seems a bit excessive.
> 
> I would say it's unusual, but not necessarily unexpected.
> 
> You have to understand (a) how C malloc works, (b) under what
> conditions memory may be given back to the OS, and (c) whether it's
> even necessary to give back memory to the OS.
> 
> Now (a) depends on what malloc implementation you're using.  We can
> assume it's Linux glibc, although even that has changed several times,
> so it really depends on which precise version of glibc you've got, but
> for this discussion I'll assume it's the latest version.  All of these
> details could be completely different for other operating systems ...
> 
> glibc currently has three strategies to allocate memory.
> 
> . For small amounts (under 512 bytes in the current impl) it has a
>  linked list of cached blocks of fixed sizes that are used to satisfy
>  requests quickly.
> 
> . For medium amounts (512 - 128K, adjustable) it has a complex
>  algorithm described as a combination of best fit and LRU.
> 
> . For large allocations (128K and over, but tunable), it uses mmap.
> 
> Furthermore, for allocations < 128K, when more core is required from
> the OS, it will either use sbrk(2) to increase the heap linearly, or
> it will use mmap(2) to allocate >= 1MB chunks scattered around the
> address space.
> 
> Basically what this means for (b) is that it's phenomenally hard to
> predict if it will be possible to give back memory to the OS.  It
> depends on how the OCaml runtime requested it (what size, what order
> of requests).  It will depend on how random C allocations (libraries
> and the OCaml runtime) happen to be spread around, since those cannot
> be moved and will prevent memory from being given back.  And it will
> depend on the malloc control structures themselves which also cannot
> be moved and their location will be highly dependent on the order in
> which requests were made (maybe even not predictable if you have a
> multithreaded program).  It may be that just one struct is preventing
> a whole mmapped area from being given back.
> 
> So you might think that your program "just allocated 1GB of RAM and
> freed it" at the OCaml level, but what's happening at the allocator
> level is likely to be far more complex.
> 
> And that brings us to (c): does it even make sense to give back memory
> to the OS?  Here's the news: the OS doesn't need you to give back
> memory.  Because of virtual memory and swap, the OS will quite happily
> take back your memory whenever it wants without asking you.  It could
> even be more efficient this way.
> 
> Rich.
> 
> -- 
> Richard Jones
> Red Hat