[Caml-list] Nproc: process pools for OCaml (request for suggestions)

[Caml-list] Nproc: process pools for OCaml (request for suggestions)

[Martin Jambon]

I would like to publicize Nproc, which is an implementation of process
pools for OCaml based on fork, pipes, Marshal and Lwt:

  https://github.com/MyLifeLabs/nproc

Using Nproc involves:

1. Creating a pool of N processes, N being chosen by the user.
2. Running tasks:
  a. Submitting a task (f, x) of any type.
  b. Defining what to do when the result becomes available.

Possible uses of Nproc include:

- running CPU-intensive tasks on multiple cores
- detaching synchronous operations for which a non-blocking version is
not available


Let me know of your comments, suggestions, questions, etc.


Martin

Re: [Caml-list] Nproc: process pools for OCaml (request for suggestions)

[Jerome Vouillon]

Hi,

On Tue, Nov 29, 2011 at 06:54:46PM -0800, Martin Jambon wrote:
> I would like to publicize Nproc, which is an implementation of process
> pools for OCaml based on fork, pipes, Marshal and Lwt:
> 
>   https://github.com/MyLifeLabs/nproc
> 
> Using Nproc involves:
> 
> 1. Creating a pool of N processes, N being chosen by the user.
> 2. Running tasks:
>   a. Submitting a task (f, x) of any type.
>   b. Defining what to do when the result becomes available.

Marco Danelutto and Roberto Di Cosmo have written a small library to
perform parallel maps and folds on multi-core machines. This is
complementary to your library. You should look at the implementation:
communication is performed by marshalling to a shared memory area for
better performances, and pipes are used only for synchronization.

     https://gitorious.org/parmap/


I have also written a more low-level library, where you can control
which process runs each task. This is useful when the processes have
to work with a lot of data that you don't want to duplicate. You can
get the code with the following command:

     darcs clone http://www.pps.jussieu.fr/~vouillon/coinst/darcs/dev/

The file of interest is task.ml.

-- Jerome

Re: [Caml-list] Nproc: process pools for OCaml (request for suggestions)

[Martin Jambon]

On 11/30/2011 01:35 AM, Jerome Vouillon wrote:
> Hi,
> 
> On Tue, Nov 29, 2011 at 06:54:46PM -0800, Martin Jambon wrote:
>> I would like to publicize Nproc, which is an implementation of process
>> pools for OCaml based on fork, pipes, Marshal and Lwt:
>>
>>   https://github.com/MyLifeLabs/nproc
>>
>> Using Nproc involves:
>>
>> 1. Creating a pool of N processes, N being chosen by the user.
>> 2. Running tasks:
>>   a. Submitting a task (f, x) of any type.
>>   b. Defining what to do when the result becomes available.
> 
> Marco Danelutto and Roberto Di Cosmo have written a small library to
> perform parallel maps and folds on multi-core machines. This is
> complementary to your library. You should look at the implementation:
> communication is performed by marshalling to a shared memory area for
> better performances, and pipes are used only for synchronization.
>
>      https://gitorious.org/parmap/
>
>

Thank you, this is interesting. I hadn't look at the implementation
although I knew about parmap but wanted something with an Lwt-ready
interface. I also wanted something that could handle continuous streams,
without having to create a process for each stream item (assuming fork()
is more expensive than we want).


> I have also written a more low-level library, where you can control
> which process runs each task. This is useful when the processes have
> to work with a lot of data that you don't want to duplicate. You can
> get the code with the following command:
> 
>      darcs clone http://www.pps.jussieu.fr/~vouillon/coinst/darcs/dev/
> 
> The file of interest is task.ml.

I see. Thanks.


Martin