Memory cost of a compaction

Memory cost of a compaction

[David Baelde]

Hi,

I have a program which memory consumption slowly increases (say from
6M to 8M resident set size, the total size being around 67M), and then
at some point after like 10 hours, suddenly exploses to 100M (rss) and
more until it crashes because of the lack of memory.

I've been tracking the problem for days. I found some memory leaks in
C bindings, but it doesn't explain the explosion which indeed still
happens. Using ocaml-memprof I could see that the number and size of
values is normal.

But I'm now wondering if after 10 hours a compaction could be
triggered because of fragmentation. I believe that a compaction would
allocate some memory in order to move data around, could it be the
cause for my explosion ? My question is: in the worst case, how much
memory can be needed for compaction ?

Thanks for any details about compaction -- pr any idea about this explosion.
-- 
David

PS: I know that I can trigger compaction manually, and log compactions
-- heap extensions are interesting too. I'll do that in the next run.

Re: [Caml-list] Memory cost of a compaction

[Hendrik Tews]

   But I'm now wondering if after 10 hours a compaction could be
   triggered because of fragmentation. I believe that a compaction would

Have you tried logging GC compatations:

  { (GC.get()) with GC.verbose <- 0x010 }

Or triggeringg compactations yourself GC.compact?

Bye,

Hendrik

Re: [Caml-list] Memory cost of a compaction

[David Baelde]

Hi,

I actually started logging Gc compactions and a few other messages
after that mail. It showed me that many compactions happened before 10
hours of run, without any noticeable cost in memory consumption.
Besides, the OCaml heap is really small compared to the C heap.

Using mtrace() I actually found out that my bug is related to vorbis
(probably used in a wrong way in ocaml-vorbis in case of downsampling)
which after ten hours requires two 50MB PCM buffers...

Anyway, after a little thinking it seems to me that not much extra
memory is needed by compactions, you can just move things inside down
the heap without extra room, right?

David