[Caml-list] library/framework needed for distributed programming in OCaml

[Caml-list] library/framework needed for distributed programming in OCaml

[Francois Berenger]

Dear list,

Please forgive me to open this possibly re-occurring thread
but I need an up to date version of the answer.

I once wrote a parallel and distributed toy program
in Python which proved to be quite useful (
http://savannah.nongnu.org/projects/par/
).
It was quite simple and easy to write because I used
a "library that enables you to build applications in which objects can 
talk to each other over the network" (http://pypi.python.org/pypi/Pyro4).

So, I am looking for the gold standard to write
modules in OCaml that can talk to each other over the network
(not objects, I don't like them so much).

Here are some requirements, in a random order:

- the target execution environment is composed of
   about 10 Linux workstations. It may switch to 1 or
   2 interconnected clusters in the future (about 512 cores max).
   So, not as large a scale as a company doing big data.
- the system will be used to transfer files of various sizes
   (big files like a few Gb included, tiny ones also)
- pure OCaml code, so JoCaml and CamlP3l are out.
   I don't like so much if there is some C part in the library
   but this is not a show stopper.
- I really dislike syntax extensions (or things that force
   me to do a lot of sysadmin strange configuration) so user-land only
   would be great
- preserving type-safety and abstraction as mentioned
   in the Quicksilver/OCaml paper would be cool (
   www.kb.ecei.tohoku.ac.jp/~sumii/pub/qs.pdf
   ) but not mandatory.
   Ideally, encryption or compression of communications should
   be a user-toggable feature.
- tolerance to partial failures would be nice but not
   mandatory (because my target environment is not so error prone
   and not so large)
- the project should be actively maintained and preferably used
   in production somewhere ("je n'aime pas essuyer les platres")
- I don't like to use huge frameworks/libraries (j'essaye d'eviter "les
   usines a gaz")

I would be satisfied with even just links to things that
satisfy most of my requirements.

For the moment, the few things that I could find that looks useful are:
  - Client-server part of
    http://caml.inria.fr/pub/docs/oreilly-book/html.bak/book-ora187.html
  - maybe the SunRPC part of
    http://projects.camlcity.org/projects/ocamlnet.html

Thanks a lot for your suggestions,
Francois.

Re: [Caml-list] library/framework needed for distributed programming in OCaml

[Lauri Alanko]

Hi.

I also had need of such a library when I started working on a  
distributed programming project, but as I couldn't find anything that  
fulfilled my requirements, I wrote my own.

The library is Lwt-based, and provides pretty straightforward remotely  
callable procedures:

type ('a, 'r) handle
val ($) : ('a, 'r) handle -> 'a -> 'r Lwt.t
val publish : ('a -> 'r Lwt.t) -> ('a, 'r) handle

The library is designed to work in a firewalled environment:  
firewalled nodes can join the distributed network by connecting to any  
accessible host of that network with an open listener. All messages  
between nodes are routed through the network.

type connection
val connect :
     ?host:string -> ?port:int -> unit -> connection Lwt.t
val listen :
     ?port:int -> unit -> unit Lwt.t

Initial values are obtained with "roots". A root (usually a handle, or  
a record or first-class module containing handles) is keyed to a  
string, and once set by a node, it can be discovered by any node in  
the network:

val get_root : string -> 'a Lwt.t
val set_root : string -> 'a -> unit Lwt.t

This is the only unsafe part of the interface: the getter and setter  
_must_ agree on the type of the root or all hell breaks loose.

I haven't published the library yet, since I have wanted to feel free  
to modify it as required by my application, but if you are interested,  
and cannot find a more mature solution, I'd be happy to provide it for  
your inspection.

Cheers,


Lauri

Re: [Caml-list] library/framework needed for distributed programming in OCaml

[Francois Berenger]

Hello,

For the moment, the best answer I got privately in my mailbox
is to combine Janestreet's bin_prot and async libraries
or Martin Jambon's biniou with Lwt.

I kind of like what I read on async and bin_prot.

On 07/11/2012 07:21 PM, Lauri Alanko wrote:
> Hi.
>
> I also had need of such a library when I started working on a
> distributed programming project, but as I couldn't find anything that
> fulfilled my requirements, I wrote my own.
>
> The library is Lwt-based, and provides pretty straightforward remotely
> callable procedures:
>
> type ('a, 'r) handle
> val ($) : ('a, 'r) handle -> 'a -> 'r Lwt.t
> val publish : ('a -> 'r Lwt.t) -> ('a, 'r) handle
>
> The library is designed to work in a firewalled environment: firewalled
> nodes can join the distributed network by connecting to any accessible
> host of that network with an open listener. All messages between nodes
> are routed through the network.
>
> type connection
> val connect :
>      ?host:string -> ?port:int -> unit -> connection Lwt.t
> val listen :
>      ?port:int -> unit -> unit Lwt.t
>
> Initial values are obtained with "roots". A root (usually a handle, or a
> record or first-class module containing handles) is keyed to a string,
> and once set by a node, it can be discovered by any node in the network:
>
> val get_root : string -> 'a Lwt.t
> val set_root : string -> 'a -> unit Lwt.t
>
> This is the only unsafe part of the interface: the getter and setter
> _must_ agree on the type of the root or all hell breaks loose.
>
> I haven't published the library yet, since I have wanted to feel free to
> modify it as required by my application, but if you are interested, and
> cannot find a more mature solution, I'd be happy to provide it for your
> inspection.

If you publish your code somewhere (github is really nice), I would
be very happy to have a look at it.

Thanks a lot,
F.