[Caml-list] strategies to deal with huge in-memory "object" graphs?

[Yoann Padioleau <pad@fb.com>]

Hi list,

I have an application that is gradually creating a graph (using ocamlgraph) and 
the amount of memory it is using is around 3 or 4 Gb (my machine has 
74Gb of RAM). There are lots of nodes and edges in this graph. The problem is that building
this graph takes a huge amount of time. As the build progresses, it gets slower
and slower. My guess is that the “object” graph is getting really huge and
so the Gc needs to explore each time even more. I’ve tried things like

  (* see www.elehack.net/michael/blog/2010/06/ocaml-memory-tuning *)
  Gc.set { (Gc.get()) with Gc.minor_heap_size = 4_000_000 };
  (* goes from 5300s to 3000s for building db for www *)
  Gc.set { (Gc.get()) with Gc.major_heap_increment = 8_000_000 };
  Gc.set { (Gc.get()) with Gc.space_overhead = 300 };


but it does not really help. It is still really slow.

In the past I sometimes use the Marshall module to reduce the number of “objects”,
but it forces me to rewrite quite a lot the code. 

Is there a way to partition the heap so that for instance in my case all the graph
related things are put in a different area that the Gc does not have to explore each time.
I’d like a minor heap, major heap, and then  a do_not_gc_this_heap_it_is_only_growing_there_is_no_garbage_here_to_collect.