Re: [Caml-list] Stopping and continuing GC

[Richard Jones <rich@annexia.org>]

On Fri, Aug 17, 2007 at 10:05:41AM -0400, Markus Mottl wrote:
> This, btw., raises an interesting question: are there any ways (and
> hopefully even intentions) to make compaction work in slices like the
> major GC does?  We write extremely latency sensitive applications and
> often even have to manually control when the GC performs collections.
> 
> Unfortunately, compaction cannot be controlled in the same way as the
> major GC can be (e.g. calling the major GC with a specific slice
> size).  That's why we have to turn off automatic compactions, too, and
> run these latency sensitive applications on machines with very large
> amounts of memory.

I looked at the code in byterun/compact.c (I've been looking at the
internals of the GC quite a bit this week) and at first glance it
seems like it overwrites headers in the heap temporarily while
compacting.  So any incremental compactor looks like it would involve
quite a rewrite.

But I wonder in general terms about the problem you are having, which
is surely a fragmentation issue (exactly the same as you'd see in a C
program).  Can this be cured by examining what sizes of structures you
are allocating?  Perhaps you can pad out or trim common structures so
that they are the same size, thus avoiding fragmentation?  A second
possibility is that it's an interaction between the OCaml heap blocks
and the underlying C malloc allocator.  It might be that OCaml is
requesting allocations which always require more memory from the
operating system (ie. they are too large or awkwardly sized to satisfy
from free areas within the current C heap).  It should be relatively
easy to diagnose either situation: For the first you'd need a program
to walk the Caml heap looking at the sizes of blocks, which you'd run
in a process that has been running for some time.  For the second,
instrument calls to malloc (in caml_alloc_for_heap).

That's just from a cursory look at the code anyway -- there is
probably much that I'm missing, but instrumenting those would be a
start.

Rich.

-- 
Richard Jones
Red Hat