Re: [Caml-list] Understanding usage by the runtime

[orbitz@ezabel.com]

Hello everyone!

I would like to reassure you that all is right in the world.  After a large number of tests I finally tracked the problem down to an entry in a Hashtbl not being deleted.  It was a one line fix!

One question does remain though: In my tests I would do some work that would cause ~1GB of RAM to be under control of the Gc.  Then I would do something that, at the time I didn't understand, would case the Gc to compact all of its memory and go back down to less than 1 meg, but the RES value in top would only drop to about 400 megs.  Is this expected behavior?  I know the malloc implementation might hold on to some data for itself but 400x the amount of memory the Ocaml RTS actually needs seems a bit excessive.  I know there is a bug report floating around from Martin about large Arrays not being properly freed, in this case my issue was with a Hashtbl.  I do not know if Hashtbl is implemented with an Array underneath, but could that be the cause of my overhead if so?

Thank you


On Jan 1, 2012, at 7:44 AM, Richard W.M. Jones wrote:

> On Sat, Dec 31, 2011 at 10:33:19AM -0500, orbitz@ezabel.com wrote:
>> Being on the C side is not even something I had considered.  In this
>> case, I think the only piece of code not part of the Ocaml RTS that is
>> talking to C is Lwt.  It is possible that there is a memory leak in
>> there somewhere.  The upside, though, is there seems to be some
>> residue of it in the Ocaml side.  My heap numbers given earlier are
>> ~65megs which is significantly larger than it should be, so I might be
>> able to track it down from the Ocaml side.
> 
> A couple of other ideas:
> 
> Is compaction disabled?  lablgtk disables it unconditionally by
> setting the global Gc max_overhead (see also the Gc documentation):
> 
>  src/gtkMain.ml:
>    let () = Gc.set {(Gc.get()) with Gc.max_overhead = 1000000}
> 
> If something in your program or Lwt does the same, you may get
> fragmentation of the C malloc heap or perhaps the OCaml heap.  I've
> experienced fragmentation in very long-running C programs and it's
> insidious because it's very hard to understand what's really going on,
> and impossible IME to remedy it.
> 
> Second suggestion is to look at /proc/<pid>/maps and/or smaps.
> That'll tell you without doubt where the 2GB of memory is being used.
> Most likely in the heap from the way you describe it, but it is worth
> checking that top isn't reporting something innocuous such as a big
> file-backed mmap in one of your C libraries.
> 
> Attached is a script that you can adapt to help you interpret
> /proc/<pid>/maps.
> 
> Rich.
> 
> -- 
> Richard Jones
> Red Hat
> <maps.pl>