Re: [Caml-list] Understanding usage by the runtime

[orbitz@ezabel.com]

In this case, as far as I know, all of the memory I was creating was under Ocaml, not a C extension which is why I would have expected the memory to be given back to malloc which would then give it back to the OS.  I understand that the malloc implementation might decide to retain some data, the overhead of 400x the amount of active data just startled me and I feel like something else was going on.  I used your maps.pl script and what the runtime seemed to be doing is growing the anonymous mapped region and then moving it into the heap and shrinking anonymous mapped region, so in my case the heap grew to 1 Gig then down to 450 megs as teh Gc could finally free the data.  I don't know what was really going on under the hood though and unfortunately not sure how to figure it out.  Thankfully, at this point it's just a curiosity not a production problem.

Thank you


On Jan 8, 2012, at 6:02 PM, Richard W.M. Jones wrote:

> On Sun, Jan 08, 2012 at 05:50:40PM -0500, orbitz@ezabel.com wrote:
>> Isn't the goal of compaction to keep all of these blocks of memory
>> as close as possible?  I should have noted the fragmentation of my
>> heap after compaction, but it seems unlikely that my < 1meg of actual
>> data could be fragmented across 400megs worth of chunks.
> 
> I might not have been clear: memory can only be given back to the OS
> at the C / malloc allocator level.  OCaml compaction has nothing to do
> with this because C allocations (and data structures used by malloc
> itself) cannot ever be moved.
> 
> However you can get a clearer picture if you look at /proc/<pid>/maps
> or smaps and also if you have a debugging malloc implementation.
> 
>>> Here's the news: the OS doesn't need you to give back
>>> memory.  Because of virtual memory and swap, the OS will quite happily
>>> take back your memory whenever it wants without asking you. 
>> 
>> For most cases this is true, however in my case (which is not the
>> usual case), my OS has no swap.  We actually prefer things to fail
>> than to be swapped because we are doing computations could take months
>> if we get into a swapping situation.  I'm no linux expert so our
>> solution is to not have swap and to keep our VMs light when it comes
>> to I/O.  Perhaps this is a poor solution but it does change things for
>> us.
> 
> Sure, this is a perfectly valid case, we have many customers who use
> RHEL like this.
> 
> Rich.
> 
> -- 
> Richard Jones
> Red Hat