Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Romain Bardou <romain.bardou@inria.fr>]

Le 19/07/2013 16:21, Török Edwin a écrit :
> On 07/19/2013 04:05 PM, Romain Bardou wrote:
>> Hello,
>>
>> I plan on writing yet another library to help with concurrency in OCaml.
>> The motivations for this library, and the interface I have in mind, are
>> available here:
>>
>> http://romain.bardou.fr/procord/api/Procord.html
>>
>> Before actually implementing the library, I would be very happy to
>> receive feedback. I am interested to know, among others:
>> - whether I miss important information which would make the very
>> existence of this library stupid (such as, it already exists);
>> - whether I forgot some important use case;
> 
> Processing streams of data in parallel without having to read all the data in memory first.

Good point. I'm not sure how to deal with this and I actually think it
calls for a slightly different paradigm. Although maybe one could
eventually add send/receive functions to send/receive to/from an 'a
process, and the same for the worker. It would allow to transmit more
than one input and more than one output.

> You could sort of do this with your current interface but only on Unix (i.e. mkfifo and pass filename).
> Don't know what'd work on Windows, perhaps creating a (named) pipe, and sending the file descriptor to a newly spawned child?

I don't think there are named pipes on Windows but I may be wrong. Maybe
it can be emulated using files anyway. I don't think sending a file
descriptor is a good idea but sending a file name is of course possible.

> Also it'd be nice to have something similar to the reduce in map-reduce.

That's definitely something I want to be able to add easily. It won't be
in the initial version, as I have no need for it right now, but it is
important for me that it can easily be added, maybe as a new layer on
top of Procord itself.

>> - whether the names I chose have better ubiquitous equivalents;
>> - whether you believe I should choose another interface entirely, for
>> instance, if you don't like functors.
> 
> I'd prefer if there was also a Lwt-like monadic interface, but it is not absolutely required (it could probably be done on top of your already existing interface).

That is also something I want to be able to add easily, but I don't want
Procord to depend on anything and to force one library. The user should
be able to choose Async or Lwt or whatever. This is why I propose
get_file_descriptors and process_status. It should be easy to write a
Lwt using that.

It would be possible to provide separate modules Procord_lwt and
Procord_async as interfaces though.

Thanks for your inputs!

Best regards,

-- 
Romain Bardou