Re: [Caml-list] Regarding SMP computing

[Richard Jones <rich@annexia.org>]

On Tue, Sep 26, 2006 at 10:37:27AM -0400, Markus Mottl wrote:
> On 9/26/06, Damien Doligez <damien.doligez@inria.fr> wrote:
> >The GC doesn't work by checking whether something has become
> >unreachable.  It works by marking all reachable data, then deallocating
> >everything else.  There is no way to do what you want in OCaml, and
> >I don't think it can be done without a major rework of the runtime
> >system.
> 
> But isn't it true that the GC doesn't follow pointers that point
> outside the OCaml-heap?  In that case it might be conceivable to copy
> OCaml-data that must not be reclaimed into the C-heap.  Of course,
> this would mean that pointers must not point back into the OCaml-heap
> from there, because there is no way the GC could know then that some
> value in the OCaml-heap is still in use, or how to update the pointer
> in the C-heap in case the OCaml-value gets moved around, e.g. during a
> compaction.
> 
> If the above really works, I'd be glad to know whether there is
> already functionality to copy OCaml-structures around.  We have some
> applications that would greatly benefit from this feature, too: they
> require an enormous amount of static background knowledge, and have to
> use it for small jobs which can be easily distributed.  We have
> multi-core, multi-processor machines, and would be able to greatly
> speed up our compute jobs if we could put these large OCaml-values
> into a shared-memory region.  It would save us both a hell lot of
> memory (probably in the range of GBs per machine for some jobs), and
> also make the GC work less hard marking data that is not going to be
> reclaimed anyway.

Really similar case to us!

In theory you'd have a "MARK_ANCIENT (obj)" operation.  It would copy
obj and everything pointed to by obj out of the OCaml heap into
malloc'd memory.  It would need to find everything pointing to obj and
update those pointers to the out-of-heap copy, or else have a proxy in
place of obj.  All done in C so that the garbage collector wouldn't be
running while this happened, and in Markus's case you'd want to move
out to some sort of shared memory, perhaps using something like
mmalloc[1].

The problem with all this is that such an "ancient" object had really
better not be changed.  If it was changed such that part of it pointed
to a new object allocated on the Caml heap, then that new object could
never be reached by the GC, and so would quickly get collected,
resulting in a dangling pointer.  You'd have to ensure that the
ancient object was never changed by careful programming ...

Rich.

[1] http://www.math.utah.edu/docs/info/mmalloc_1.html

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Internet Marketing and AdWords courses - http://merjis.com/courses - NEW!
Merjis blog - http://blog.merjis.com - NEW!