[Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Romain Bardou]

Hello,

I plan on writing yet another library to help with concurrency in OCaml.
The motivations for this library, and the interface I have in mind, are
available here:

http://romain.bardou.fr/procord/api/Procord.html

Before actually implementing the library, I would be very happy to
receive feedback. I am interested to know, among others:
- whether I miss important information which would make the very
existence of this library stupid (such as, it already exists);
- whether I forgot some important use case;
- whether the names I chose have better ubiquitous equivalents;
- whether you believe I should choose another interface entirely, for
instance, if you don't like functors.

Thanks and cheers,

-- 
Romain Bardou

Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Török Edwin]

On 07/19/2013 04:05 PM, Romain Bardou wrote:
> Hello,
> 
> I plan on writing yet another library to help with concurrency in OCaml.
> The motivations for this library, and the interface I have in mind, are
> available here:
> 
> http://romain.bardou.fr/procord/api/Procord.html
> 
> Before actually implementing the library, I would be very happy to
> receive feedback. I am interested to know, among others:
> - whether I miss important information which would make the very
> existence of this library stupid (such as, it already exists);
> - whether I forgot some important use case;

Processing streams of data in parallel without having to read all the data in memory first.
You could sort of do this with your current interface but only on Unix (i.e. mkfifo and pass filename).
Don't know what'd work on Windows, perhaps creating a (named) pipe, and sending the file descriptor to a newly spawned child?

Also it'd be nice to have something similar to the reduce in map-reduce.

> - whether the names I chose have better ubiquitous equivalents;
> - whether you believe I should choose another interface entirely, for
> instance, if you don't like functors.

I'd prefer if there was also a Lwt-like monadic interface, but it is not absolutely required (it could probably be done on top of your already existing interface).

Best regards,
--Edwin

Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Romain Bardou]

Le 19/07/2013 16:21, Török Edwin a écrit :
> On 07/19/2013 04:05 PM, Romain Bardou wrote:
>> Hello,
>>
>> I plan on writing yet another library to help with concurrency in OCaml.
>> The motivations for this library, and the interface I have in mind, are
>> available here:
>>
>> http://romain.bardou.fr/procord/api/Procord.html
>>
>> Before actually implementing the library, I would be very happy to
>> receive feedback. I am interested to know, among others:
>> - whether I miss important information which would make the very
>> existence of this library stupid (such as, it already exists);
>> - whether I forgot some important use case;
> 
> Processing streams of data in parallel without having to read all the data in memory first.

Good point. I'm not sure how to deal with this and I actually think it
calls for a slightly different paradigm. Although maybe one could
eventually add send/receive functions to send/receive to/from an 'a
process, and the same for the worker. It would allow to transmit more
than one input and more than one output.

> You could sort of do this with your current interface but only on Unix (i.e. mkfifo and pass filename).
> Don't know what'd work on Windows, perhaps creating a (named) pipe, and sending the file descriptor to a newly spawned child?

I don't think there are named pipes on Windows but I may be wrong. Maybe
it can be emulated using files anyway. I don't think sending a file
descriptor is a good idea but sending a file name is of course possible.

> Also it'd be nice to have something similar to the reduce in map-reduce.

That's definitely something I want to be able to add easily. It won't be
in the initial version, as I have no need for it right now, but it is
important for me that it can easily be added, maybe as a new layer on
top of Procord itself.

>> - whether the names I chose have better ubiquitous equivalents;
>> - whether you believe I should choose another interface entirely, for
>> instance, if you don't like functors.
> 
> I'd prefer if there was also a Lwt-like monadic interface, but it is not absolutely required (it could probably be done on top of your already existing interface).

That is also something I want to be able to add easily, but I don't want
Procord to depend on anything and to force one library. The user should
be able to choose Async or Lwt or whatever. This is why I propose
get_file_descriptors and process_status. It should be easy to write a
Lwt using that.

It would be possible to provide separate modules Procord_lwt and
Procord_async as interfaces though.

Thanks for your inputs!

Best regards,

-- 
Romain Bardou

Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Török Edwin]

On 07/19/2013 05:35 PM, Romain Bardou wrote:
> Le 19/07/2013 16:21, Török Edwin a écrit :
>> On 07/19/2013 04:05 PM, Romain Bardou wrote:
>>> Hello,
>>>
>>> I plan on writing yet another library to help with concurrency in OCaml.
>>> The motivations for this library, and the interface I have in mind, are
>>> available here:
>>>
>>> http://romain.bardou.fr/procord/api/Procord.html
>>>
>>> Before actually implementing the library, I would be very happy to
>>> receive feedback. I am interested to know, among others:
>>> - whether I miss important information which would make the very
>>> existence of this library stupid (such as, it already exists);
>>> - whether I forgot some important use case;
>>
>> Processing streams of data in parallel without having to read all the data in memory first.
> 
> Good point. I'm not sure how to deal with this and I actually think it
> calls for a slightly different paradigm. Although maybe one could
> eventually add send/receive functions to send/receive to/from an 'a
> process, and the same for the worker. It would allow to transmit more
> than one input and more than one output.

Yeah I think it'd need a concept that feeding input to a worker won't necessarely produce an immediate output (for example: compressing a file). And you'd need a way to signal EOF. 

I don't know how this could be done in a generic way (for an 'a), but maybe something could be done for strings: provide the worker with a way to repeatedly read strings (like Pervasives.input), and to repeatedly write output (like Pervasives.output). 
If you can use pipes then they can even be Pervasives.input and Pervasives.output themselves, otherwise
some buffering logic will be needed.

The nice thing about this is that you could quite easily turn something like Cryptokit's transform into a worker, even if you are reading/writing from/to the network.

> 
>> You could sort of do this with your current interface but only on Unix (i.e. mkfifo and pass filename).
>> Don't know what'd work on Windows, perhaps creating a (named) pipe, and sending the file descriptor to a newly spawned child?
> 
> I don't think there are named pipes on Windows but I may be wrong. Maybe
> it can be emulated using files anyway. I don't think sending a file
> descriptor is a good idea but sending a file name is of course possible.
> 
>> Also it'd be nice to have something similar to the reduce in map-reduce.
> 
> That's definitely something I want to be able to add easily. It won't be
> in the initial version, as I have no need for it right now, but it is
> important for me that it can easily be added, maybe as a new layer on
> top of Procord itself.
> 
>>> - whether the names I chose have better ubiquitous equivalents;
>>> - whether you believe I should choose another interface entirely, for
>>> instance, if you don't like functors.
>>
>> I'd prefer if there was also a Lwt-like monadic interface, but it is not absolutely required (it could probably be done on top of your already existing interface).
> 
> That is also something I want to be able to add easily, but I don't want
> Procord to depend on anything and to force one library. The user should
> be able to choose Async or Lwt or whatever. This is why I propose
> get_file_descriptors and process_status. It should be easy to write a
> Lwt using that.
> 
> It would be possible to provide separate modules Procord_lwt and
> Procord_async as interfaces though.
> 
> Thanks for your inputs!
> 
> Best regards,
> 

Re: [Caml-list] Request for feedback: Procord, a library to delegate tasks to other processes

[Francois Berenger]

On 07/19/2013 11:35 PM, Romain Bardou wrote:
> Le 19/07/2013 16:21, Török Edwin a écrit :
>> On 07/19/2013 04:05 PM, Romain Bardou wrote:
>>> Hello,
>>>
>>> I plan on writing yet another library to help with concurrency in OCaml.
>>> The motivations for this library, and the interface I have in mind, are
>>> available here:
>>>
>>> http://romain.bardou.fr/procord/api/Procord.html
>>>
>>> Before actually implementing the library, I would be very happy to
>>> receive feedback. I am interested to know, among others:
>>> - whether I miss important information which would make the very
>>> existence of this library stupid (such as, it already exists);
>>> - whether I forgot some important use case;
>>
>> Processing streams of data in parallel without having to read all the data in memory first.

Use a lazy list.

>[...]