Re: [Caml-list] Shared data space: How to manage Ancient keys?

[Goswin von Brederlow <goswin-v-b@web.de>]

Hugo Ferreira <hmf@inescporto.pt> writes:

> Hello,
>
> I am trying to implement a parallel algorithm and have opted to use
> the Ancient module to share data across processes because it dynamically
> reallocates memory when mapping data (I don't know the size of the
> objects before hand). However I am having trouble managing its use of
> keys.
>
> Maybe someone here has a solution to my problem.
>
> To make things clear - a little background. I have a set of worker
> processes that cooperate to analyse a set of sequences. I assume
> all sequences are in a global blackboard-type /LINDA-like data space.
> Each worker removes a single sequence. It then scans all other existing
> sequences and selects another one. It then a) removes the selected
> sequence b) merges the 2 sequences at hand and generates 0, 1 or more
> new sequences. The process repeats itself until no more sequences exist
> in the global data space.
>
> In the above scenario all sequences would be assigned a unique id which
> increases sequentially. If I used a common data structure such as a
> binary tree I could simply increment a counter and use that as a key
> for the insertion/deletion of sequences into/from the binary tree.
> However I cannot do this here because I can only share linear memory
> via mapped files in Ocaml.
>
> So far I have made a small experiment that allows me to insert data
> into a shared memory space. Ancient allows one to copy various objects
> into the memory space via an integer index. From the code it seems that
> the indexing table is an array like structure. So when I add and object
> I must identify an index (slot) that is empty (unused) and reuse that
> otherwise memory will grow unchecked.
>
> My question is how can I generate and share keys for Ancient's use
> without exceeding memory usage?
>
> TIA,
> Hugo F.

How about a simple solution. Only one thread handles indexes.

You said you have worker processes. Add one controling process on top of
it that spawns all the worker processes and keeps a communication line
open with them, e.g. a pipe. Each worker would then tell the controller
to remove an index or request an unused index to store a result. Its
probably a good idea to request unused indexes in bunches, say 100 at a
time to cut down on latenzy.

MfG
        Goswin