[Caml-list] Distributed computing libraries

[Caml-list] Distributed computing libraries

[Thomas Braibant]

Hi list,

There exists several different "distributed computing libraries" in OCaml:

- Functory http://functory.lri.fr/
- JoCaml http://jocaml.inria.fr/ (not really a library, though)
- Nproc https://github.com/MyLifeLabs/nproc
- OCaml for multicore http://www.algo-prog.info/ocmc/web/ (not a library either)
- Parmap https://gitorious.org/parmap/parmap
- Plasma Map/Reduce http://plasma.camlcity.org/plasma/

And I am pretty sure that this list is not exhaustive, looking at
http://caml.inria.fr/cgi-bin/hump.en.cgi?sort=0&browse=77

Maybe it would be interesting for the community if someone could sum
up the pros and cons of each of this "libraries", and maybe give some
information about their status (still in development, mature, etc)?

With best regards,
Thomas Braibant

Re: [Caml-list] Distributed computing libraries

[oliver]

On Wed, Jun 06, 2012 at 12:53:07PM -0400, Thomas Braibant wrote:
> Hi list,
> 
> There exists several different "distributed computing libraries" in OCaml:
> 
> - Functory http://functory.lri.fr/
> - JoCaml http://jocaml.inria.fr/ (not really a library, though)
> - Nproc https://github.com/MyLifeLabs/nproc
> - OCaml for multicore http://www.algo-prog.info/ocmc/web/ (not a library either)
> - Parmap https://gitorious.org/parmap/parmap
> - Plasma Map/Reduce http://plasma.camlcity.org/plasma/
[...]

  - CamlP3l http://camlp3l.inria.fr/eng.htm


[...]
> Maybe it would be interesting for the community if someone could sum
> up the pros and cons of each of this "libraries", and maybe give some
> information about their status (still in development, mature, etc)?
[...]

Yes this would be fine.
And maybe "someone" could also mean some people...
...maybe there are people out there, that now one or two of them well,
and other of these libs not or not well.

...so that the information can be gathered by a lot of people.

But it would be fine to have some criteria for classifying.

For example:

  - one / multicore
  - local machine / distributed via net
  - threads / processes
  - just library calls, language extension, or seperate language 
  - toplevel-runtime / byte code / native-code
  - necessity of code rewrite for local and remote/distributive stuff,
     or possibility of transparent usage
  - ...


Ciao,
   Oliver

Re: [Caml-list] Distributed computing libraries

[Vincent Aravantinos]

On 06/06/2012 01:31 PM, oliver wrote:
> On Wed, Jun 06, 2012 at 12:53:07PM -0400, Thomas Braibant wrote:
>> Maybe it would be interesting for the community if someone could sum
>> up the pros and cons of each of this "libraries", and maybe give some
>> information about their status (still in development, mature, etc)?
> [...]
>
> Yes this would be fine.
> And maybe "someone" could also mean some people...
> ...maybe there are people out there, that now one or two of them well,
> and other of these libs not or not well.
>
> ...so that the information can be gathered by a lot of people.
>
> But it would be fine to have some criteria for classifying.
>
> For example:
>
>    - one / multicore
>    - local machine / distributed via net
>    - threads / processes
>    - just library calls, language extension, or seperate language
>    - toplevel-runtime / byte code / native-code
>    - necessity of code rewrite for local and remote/distributive stuff,
>       or possibility of transparent usage
- compatibility with commonly used libraries

-- 
Vincent Aravantinos
Postdoctoral Fellow, Concordia University, Hardware Verification Group
http://users.encs.concordia.ca/~vincent

AW: [Caml-list] Distributed computing libraries

[Gerd Stolpmann]

Am 06.06.2012 18:53:07 schrieb(en) Thomas Braibant:
> Hi list,
> 
> There exists several different "distributed computing libraries" in  
> OCaml:
> 
> - Functory http://functory.lri.fr/
> - JoCaml http://jocaml.inria.fr/ (not really a library, though)
> - Nproc https://github.com/MyLifeLabs/nproc
> - OCaml for multicore http://www.algo-prog.info/ocmc/web/ (not a  
> library either)
> - Parmap https://gitorious.org/parmap/parmap
> - Plasma Map/Reduce http://plasma.camlcity.org/plasma/
> 

These are very different pieces of software, because they tackle  
problems at various abstraction levels. ocmc is the most level-level  
here, as it "only" tries to improve the runtime so that threads can be  
run in parallel on multiple cores. That's it, there is no additional  
abstraction on top of the standard threading API - no distribution, no  
computing.

JoCaml uses the normal multi-threading in the runtime, but integrates  
it differently into the language. So, it adds abstraction, but you are  
still limited to a single core.

So far I know, all the other libraries base on multi-processing to run  
programs on multiple cores. Plasma is the only one with true  
distribution capabiltiies beyond a single computer, but the price is  
that you must use the map/reduce scheme, whereas functory or parmap  
leave you more freedom. However, all multi-processing approaches share  
the property that the data flow is limited by process boundaries  
(unless you go really low-level and also take Netmulticore into  
consideration (part of Ocamlnet), which uses shared memory to overcome  
these limitations).

I don't know what you are exactly looking for. Knowing the problem it  
would be easier to recommend something.

Gerd



> And I am pretty sure that this list is not exhaustive, looking at
> http://caml.inria.fr/cgi-bin/hump.en.cgi?sort=0&browse=77
> 
> Maybe it would be interesting for the community if someone could sum
> up the pros and cons of each of this "libraries", and maybe give some
> information about their status (still in development, mature, etc)?
> 
> With best regards,
> Thomas Braibant
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 
> 



-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

Re: AW: [Caml-list] Distributed computing libraries

[Edgar Friendly]

> So far I know, all the other libraries base on multi-processing to run
> programs on multiple cores. Plasma is the only one with true
> distribution capabiltiies beyond a single computer, but the price is
> that you must use the map/reduce scheme, whereas functory or parmap
> leave you more freedom.

Functory is able to run across a network, with clients listening for 
connections from the master to give them work to do.  It is restricted 
to a variant of map/reduce, so it doesn't give you that much freedom.

E.

Re: AW: [Caml-list] Distributed computing libraries

[oliver]

On Wed, Jun 06, 2012 at 02:16:51PM -0400, Edgar Friendly wrote:
> >So far I know, all the other libraries base on multi-processing to run
> >programs on multiple cores. Plasma is the only one with true
> >distribution capabiltiies beyond a single computer, but the price is
> >that you must use the map/reduce scheme, whereas functory or parmap
> >leave you more freedom.
> 
> Functory is able to run across a network, with clients listening for
> connections from the master to give them work to do.  It is
> restricted to a variant of map/reduce, so it doesn't give you that
> much freedom.


OCamlP3l allows computations transparently local or runnuning over many hosts.

Ciao,
   Oliver

AW: AW: [Caml-list] Distributed computing libraries

[Gerd Stolpmann]

Am 06.06.2012 20:16:51 schrieb(en) Edgar Friendly:
>> So far I know, all the other libraries base on multi-processing to  
>> run
>> programs on multiple cores. Plasma is the only one with true
>> distribution capabiltiies beyond a single computer, but the price is
>> that you must use the map/reduce scheme, whereas functory or parmap
>> leave you more freedom.
> 
> Functory is able to run across a network, with clients listening for  
> connections from the master to give them work to do.  It is  
> restricted to a variant of map/reduce, so it doesn't give you that  
> much freedom.

Thanks for correcting me.

As you know Functory a bit better: how is the data delivered to network  
clients? Just via the network connection that also assigns the work to  
do? That would be a bit lame, and not really useful.

Gerd


> E.
> 
>-- 
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
> 


-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

Re: AW: AW: [Caml-list] Distributed computing libraries

[Edgar Friendly]

On 06/06/2012 02:35 PM, Gerd Stolpmann wrote:
> As you know Functory a bit better: how is the data delivered to network
> clients? Just via the network connection that also assigns the work to
> do? That would be a bit lame, and not really useful.

Yes, the input parameters of the function to be run are sent over the 
network connection that assigns work to do, and the results come back 
over that same connection.  That said, it's possible to use a network 
file system to distribute the real data and just pass filenames over the 
control channel.

E.

Re: [Caml-list] Distributed computing libraries

[Thomas Braibant]

> These are very different pieces of software, because they tackle problems at
> various abstraction levels. ocmc is the most level-level here, as it "only"
> tries to improve the runtime so that threads can be run in parallel on
> multiple cores. That's it, there is no additional abstraction on top of the
> standard threading API - no distribution, no computing.
>
> JoCaml uses the normal multi-threading in the runtime, but integrates it
> differently into the language. So, it adds abstraction, but you are still
> limited to a single core.
>
> So far I know, all the other libraries base on multi-processing to run
> programs on multiple cores. Plasma is the only one with true distribution
> capabiltiies beyond a single computer, but the price is that you must use
> the map/reduce scheme, whereas functory or parmap leave you more freedom.
> However, all multi-processing approaches share the property that the data
> flow is limited by process boundaries (unless you go really low-level and
> also take Netmulticore into consideration (part of Ocamlnet), which uses
> shared memory to overcome these limitations).
>
> I don't know what you are exactly looking for. Knowing the problem it would
> be easier to recommend something.

Thanks a lot for this answer.

As I said in another email, I do not have a problem to solve right
now. But, I was tickled by the Parmap announce, and I wondered how it
fared with the other related systems I knew about. (Yet, I reckon that
I should have made the question more precise, with a detailed lists of
criteria, akin to what Oliver suggested.)

In my mind, asking on the list was the best way to make a good summary
of the current situation on this front. (But I am afraid I do not know
the best place to make public this kind of summaries.)

Re: [Caml-list] Distributed computing libraries

[Francois Berenger]

On 06/07/2012 05:52 AM, Thomas Braibant wrote:
>> These are very different pieces of software, because they tackle problems at
>> various abstraction levels. ocmc is the most level-level here, as it "only"
>> tries to improve the runtime so that threads can be run in parallel on
>> multiple cores. That's it, there is no additional abstraction on top of the
>> standard threading API - no distribution, no computing.
>>
>> JoCaml uses the normal multi-threading in the runtime, but integrates it
>> differently into the language. So, it adds abstraction, but you are still
>> limited to a single core.
>>
>> So far I know, all the other libraries base on multi-processing to run
>> programs on multiple cores. Plasma is the only one with true distribution
>> capabiltiies beyond a single computer, but the price is that you must use
>> the map/reduce scheme, whereas functory or parmap leave you more freedom.
>> However, all multi-processing approaches share the property that the data
>> flow is limited by process boundaries (unless you go really low-level and
>> also take Netmulticore into consideration (part of Ocamlnet), which uses
>> shared memory to overcome these limitations).
>>
>> I don't know what you are exactly looking for. Knowing the problem it would
>> be easier to recommend something.
>
> Thanks a lot for this answer.
>
> As I said in another email, I do not have a problem to solve right
> now. But, I was tickled by the Parmap announce,

Don't hesitate, jump on it, that's really a nice technology.
The "minimal disruption" concept is quite interesting.
For example, if your List.iter is changed to a Parmap.pariter,
the parallelization of this portion of the code is done.

It's quite comfortable to develop and debug in single core mode 
(List.iter) and switch to the // version only once you're happy with
the sequential one.

Regards,
F.

 > and I wondered how it
> fared with the other related systems I knew about. (Yet, I reckon that
> I should have made the question more precise, with a detailed lists of
> criteria, akin to what Oliver suggested.)
>
> In my mind, asking on the list was the best way to make a good summary
> of the current situation on this front. (But I am afraid I do not know
> the best place to make public this kind of summaries.)
>

Re: [Caml-list] Distributed computing libraries

[Thomas Braibant]

> Don't hesitate, jump on it, that's really a nice technology.
> The "minimal disruption" concept is quite interesting.
> For example, if your List.iter is changed to a Parmap.pariter,
> the parallelization of this portion of the code is done.
>
> It's quite comfortable to develop and debug in single core mode (List.iter)
> and switch to the // version only once you're happy with
> the sequential one.

I see. However, looking at, e.g., Functory,
http://www.lri.fr/~filliatr/functory/doc/Functory.Cores.html it seems
that I can use it in the same fashion (Using Functory.Cores.map
instead of List.map) and that I can debug my code using
Functory.Sequential.map.

So this does not really discriminate between the two libraries. Note,
however, that I cannot find Parmap's API described on line (using
ocamldoc). Right now, it seems that I have to download it, to generate
the doc.

Note that Gerd's Plasma Map/Reduce has a nice and comprehensive
documentation available, but, being more ambitious maybe, it is harder
for a beginner to find his/her way in: the entry cost is higher.

By comparison, JOcaml's manual is written in a quasi-tutorial fashion
( http://jocaml.inria.fr/doc/index.html ) which makes it more easy to
start hacking stuff, even if the scope is a bit different.

And what about CamlP3l? What is its status? Is it superseded by Parmap
(the lists of authors of the two softwares have a non-empty
intersection)?

Maybe that I should add some criteria to Oliver's list:
- active / inactive
- used in the wild
- documentation (API, manual)
- examples / tutorials

With best regards,
Thomas

Re: [Caml-list] Distributed computing libraries

[Francois Berenger]

Hello,

On 06/08/2012 12:44 AM, Thomas Braibant wrote:
>> Don't hesitate, jump on it, that's really a nice technology.
>> The "minimal disruption" concept is quite interesting.
>> For example, if your List.iter is changed to a Parmap.pariter,
>> the parallelization of this portion of the code is done.
>>
>> It's quite comfortable to develop and debug in single core mode (List.iter)
>> and switch to the // version only once you're happy with
>> the sequential one.
>
> I see. However, looking at, e.g., Functory,
> http://www.lri.fr/~filliatr/functory/doc/Functory.Cores.html it seems
> that I can use it in the same fashion (Using Functory.Cores.map
> instead of List.map) and that I can debug my code using
> Functory.Sequential.map.

Functory can distribute computations while parmap can only parallelize 
them on your local machine.
I'm pretty sure that parmap is faster at doing what they can be compared 
on because it has specialized optimizations that you cannot use if your 
data has to be distributed over the network.

The last time I looked at Functory:
- Functory had several not tail recursive functions and blew my stack
   (it was about one year ago, maybe things have changed)
- not any mailing list for functory
- while parmap has:
   parmap-devel <parmap-devel@inria.fr> and
   parmap-users <parmap-users@inria.fr>
- no GODI package for functory
- parmap has a GODI package (thanks to Jerome
   Maloberti) and maybe very soon an oasis one

> So this does not really discriminate between the two libraries. Note,
> however, that I cannot find Parmap's API described on line (using
> ocamldoc). Right now, it seems that I have to download it, to generate
> the doc.

At least there is a doc and you even managed to generate it.
('make doc' in a parmap checkout).
Maybe the doc will be online soon, who knows. ;)

> Note that Gerd's Plasma Map/Reduce has a nice and comprehensive
> documentation available, but, being more ambitious maybe, it is harder
> for a beginner to find his/her way in: the entry cost is higher.

It's also for distributed computing, parmap is for local parallelization.

> By comparison, JOcaml's manual is written in a quasi-tutorial fashion
> ( http://jocaml.inria.fr/doc/index.html ) which makes it more easy to
> start hacking stuff, even if the scope is a bit different.

I'm not sure it's standard OCaml code then.
At least with parmap and functory, they are just libraries that
you can plug into any plain vanilla OCaml source code.

> And what about CamlP3l? What is its status? Is it superseded by Parmap
> (the lists of authors of the two softwares have a non-empty
> intersection)?

I don't think it's just a library too.
Some more knowledgeable people may answer you on these.

Regards,
F.

Re: [Caml-list] Distributed computing libraries

[Francois Berenger]

On 06/08/2012 12:44 AM, Thomas Braibant wrote:
>> Don't hesitate, jump on it, that's really a nice technology.
>> The "minimal disruption" concept is quite interesting.
>> For example, if your List.iter is changed to a Parmap.pariter,
>> the parallelization of this portion of the code is done.
>>
>> It's quite comfortable to develop and debug in single core mode (List.iter)
>> and switch to the // version only once you're happy with
>> the sequential one.
>
> I see. However, looking at, e.g., Functory,
> http://www.lri.fr/~filliatr/functory/doc/Functory.Cores.html it seems
> that I can use it in the same fashion (Using Functory.Cores.map
> instead of List.map) and that I can debug my code using
> Functory.Sequential.map.
>
> So this does not really discriminate between the two libraries. Note,
> however, that I cannot find Parmap's API described on line (using
> ocamldoc). Right now, it seems that I have to download it, to generate
> the doc.

Roberto Di Cosmo just created a home page:
http://www.dicosmo.org/code/parmap/

The ocamldoc-generated pages are here:
http://www.dicosmo.org/code/parmap/doc/Parmap.html

Regards,
F.

> Note that Gerd's Plasma Map/Reduce has a nice and comprehensive
> documentation available, but, being more ambitious maybe, it is harder
> for a beginner to find his/her way in: the entry cost is higher.
>
> By comparison, JOcaml's manual is written in a quasi-tutorial fashion
> ( http://jocaml.inria.fr/doc/index.html ) which makes it more easy to
> start hacking stuff, even if the scope is a bit different.
>
> And what about CamlP3l? What is its status? Is it superseded by Parmap
> (the lists of authors of the two softwares have a non-empty
> intersection)?
>
> Maybe that I should add some criteria to Oliver's list:
> - active / inactive
> - used in the wild
> - documentation (API, manual)
> - examples / tutorials
>
> With best regards,
> Thomas
>

Re: [Caml-list] Distributed computing libraries

[Roberto Di Cosmo]

On Wed, Jun 06, 2012 at 12:53:07PM -0400, Thomas Braibant wrote:
<snip>
> Maybe it would be interesting for the community if someone could sum
> up the pros and cons of each of this "libraries", and maybe give some
> information about their status (still in development, mature, etc)?
> 
> With best regards,
> Thomas Braibant

Quite a good point, and it really goes beyond just the bunch of
(very useful) libraries that try to speed-up computations by
leveraging multicore/multiprocessors/clusters etc.

I think OCaml is a wonderful language, and it has the potential
to attract much more users if we can make it easy to find, build,
install, compare and assess the various libraries, packages
and tools that are available.

If I can dream with my eyes open (which is a decent way of
spending time while waiting for a plane in the middle
of Nebraska, after all :-)), I could imagine:

 (1) a central repository for OCaml projects where
   packages are uploaded, with their different versions, tagged,
   and classified, with pointers to the current, active development
   platform (github, gitorious, sourceforge, whatever) (Oasis DB?)

 (2) a continuous integration server testing buildability of the
   different versions of the packages, with different versions
   of the compiler, and on all available architectures, reporting
   errors in a meaningful way to their authors, and providing 
   extremely valuable information to users that will know 
   beforehand what works and what does not work on which
   configuration, instead of discovering this the hard way
   (Debian OCamlers lurking here surely know what is the source
    of inspiration of this part of my dream; and yes, Debian does
    all this for the Debian OCaml packages, but we need to cater
    to *all* users :-))

 (3) an organised OCamlPedia site where, for each 'class' of 
   problem (parallelism, parsing, graph algorithms, numerical
   computation, code transformation, you name it) developers and users
   can contribute to maintain up-to-date summaries, comparisons,
   test results, code snippets, pointers to scientific papers
   and the like...
   And for each "feature" of the language, I'd love to see tutorials,
   code examples, pointers to the relevant literature, and to
   packages showing them off.

To get there, we need to make our collaborative hidden side shine,
and complete the effort started by some of us to organise the community...
maybe this will happen during the coffee break at the next OCaml meeting.

Ok, time to stop dreaming, to go back to reality, and to catch that flight

--Roberto

P.S.: hey, I just checked, and ocamlpedia.* is available,
so I registered ocamlpedia.org, and I'll be more than happy
to offer it to whomever will run the site, if it happens :-)

Re: [Caml-list] Distributed computing libraries

[Thomas Braibant]

<snip>
> I think OCaml is a wonderful language, and it has the potential
> to attract much more users if we can make it easy to find, build,
> install, compare and assess the various libraries, packages
> and tools that are available.

I completely agree with what you just said, and that was the point of
my question: trying to assess, in a single thread, the  various
possibilities that exist to solve a given problem [1]. Now, it happens
that I had been thinking about the same kind of mail about package
managers, because there is also a huge variety of solutions. So, I
could ask, what are the pros and cons of the following package
managers?

- https://github.com/thelema/odb
- https://github.com/hcarty/ocamlbrew
- https://github.com/OCamlPro/opam
- http://godi.camlcity.org/godi/index.html
- http://oasis.forge.ocamlcore.org/oasis-db.html

But I do not want to fork that thread yet ;).

I think a more general perspective on this was given a while ago on
this list by Xavier, a while ago on this list (trying to paraphrase):
- the community needs a common packaging system
- there is often a lack of convergence between related projects (in
this case, libraries, for instance).

>   And for each "feature" of the language, I'd love to see tutorials,
>   code examples, pointers to the relevant literature, and to
>   packages showing them off.

On a related note, I was delighted to see that there are two books
about OCaml in the pipes currently, and I think this is great.

Thomas
[1] Actually, I did that also a while ago, about openGL bindings, and
the following discussion was quite fruitful

[Caml-list] OCaml package managers

[Francois Berenger]

On 06/07/2012 05:43 AM, Thomas Braibant wrote:
> <snip>
>> I think OCaml is a wonderful language, and it has the potential
>> to attract much more users if we can make it easy to find, build,
>> install, compare and assess the various libraries, packages
>> and tools that are available.
>
> I completely agree with what you just said, and that was the point of
> my question: trying to assess, in a single thread, the  various
> possibilities that exist to solve a given problem [1]. Now, it happens
> that I had been thinking about the same kind of mail about package
> managers, because there is also a huge variety of solutions. So, I
> could ask, what are the pros and cons of the following package
> managers?
>
> - https://github.com/thelema/odb
> - https://github.com/hcarty/ocamlbrew
> - https://github.com/OCamlPro/opam
> - http://godi.camlcity.org/godi/index.html
> - http://oasis.forge.ocamlcore.org/oasis-db.html

Here is my personal experience with some of them (ocamlbrew, odb, oasis 
and oasis-db):
- ocamlbrew can be used to bootstrap odb (get OCaml and findlib from
   scratch)
- from there, odb can install other OCaml libraries and programs
- odb can install oasis and then use oasis-db to get packages
- you can modify a 'packages' file so that odb can install things
   that are not available publicly or that are not yet packaged for oasis

 From my point of view, these tools definitely have the UNIX taste:
they are fast, they are easy to use and quite easy to troubleshoot in 
case something doesn't work as expected; which make them excellent in my 
opinion compared to other solutions.

> But I do not want to fork that thread yet ;).

Why not? ;)

Regards,
F.

Re: [Caml-list] Distributed computing libraries

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 949 bytes --]

On Wed, Jun 6, 2012 at 4:24 PM, Roberto Di Cosmo <roberto@dicosmo.org>wrote:


>   (3) an organised OCamlPedia site where


The ocamlweb project announced at last year's OCaml User's Meeting can
serve this purpose. Please see the master repo here [1] and the live site
here [2]*. We welcome and need the community to add more content. We've
attempted to make contributing easier. You just checkout a git project, add
simple html, and submit a pull request. This is perhaps not as nice as a
wiki, but it let's us scale up to other features that wikis are not
suitable for.

If you need help getting a larger section started, such as on the topic of
this thread, just email me or the mailing list [3].

[1] https://github.com/agarwal/ocamlweb
[2] http://www.ocaml-lang.org/
[3] https://lists.forge.ocamlcore.org/cgi-bin/listinfo/ocamlweb-devel
* Note the live site is behind the master repo as we have some cleaning up
yet to do in the master branch.

[-- Attachment #2: Type: text/html, Size: 1536 bytes --]

Re: [Caml-list] Distributed computing libraries

[Roberto Di Cosmo]

Thanks Ashish,
       I know the site, and it's nice it has been started.
Nonetheless, I really believe that if we want this to take
off, having people edit pages in HTML through git merge requests
will really not scale up... :-)

--Roberto

On Wed, Jun 06, 2012 at 06:23:47PM -0400, Ashish Agarwal wrote:
> On Wed, Jun 6, 2012 at 4:24 PM, Roberto Di Cosmo <roberto@dicosmo.org> wrote:
>  
> 
>      (3) an organised OCamlPedia site where
> 
> 
> The ocamlweb project announced at last year's OCaml User's Meeting can serve
> this purpose. Please see the master repo here [1] and the live site here [2]*.
> We welcome and need the community to add more content. We've attempted to make
> contributing easier. You just checkout a git project, add simple html, and
> submit a pull request. This is perhaps not as nice as a wiki, but it let's us
> scale up to other features that wikis are not suitable for.
> 
> If you need help getting a larger section started, such as on the topic of this
> thread, just email me or the mailing list [3].
> 
> [1] https://github.com/agarwal/ocamlweb
> [2] http://www.ocaml-lang.org/
> [3] https://lists.forge.ocamlcore.org/cgi-bin/listinfo/ocamlweb-devel
> * Note the live site is behind the master repo as we have some cleaning up yet
> to do in the master branch.
> 

-- 
--Roberto Di Cosmo
 
------------------------------------------------------------------
Professeur               En delegation a l'INRIA
PPS                      E-mail: roberto@dicosmo.org
Universite Paris Diderot WWW  : http://www.dicosmo.org
Case 7014                Tel  : ++33-(0)1-57 27 92 20
5, Rue Thomas Mann       
F-75205 Paris Cedex 13   Identica: http://identi.ca/rdicosmo
FRANCE.                  Twitter: http://twitter.com/rdicosmo
------------------------------------------------------------------
Attachments:
MIME accepted, Word deprecated
      http://www.gnu.org/philosophy/no-word-attachments.html
------------------------------------------------------------------
Office location:
 
Bureau 6C08 (6th floor)
175, rue du Chevaleret, XIII
Metro Chevaleret, ligne 6
------------------------------------------------------------------                                                 

Re: [Caml-list] Distributed computing libraries

[Ashish Agarwal]

[-- Attachment #1: Type: text/plain, Size: 2427 bytes --]

On Fri, Jun 8, 2012 at 2:55 AM, Roberto Di Cosmo <roberto@dicosmo.org>


>       I know the site, and it's nice it has been started.
> Nonetheless, I really believe that if we want this to take
> off, having people edit pages in HTML through git merge requests
> will really not scale up... :-)
>

Indeed, our long term goal is to have a rich website with many features,
but this was an easy way to get started quickly.



On Wed, Jun 06, 2012 at 06:23:47PM -0400, Ashish Agarwal wrote:
> > On Wed, Jun 6, 2012 at 4:24 PM, Roberto Di Cosmo <roberto@dicosmo.org>
> wrote:
> >
> >
> >      (3) an organised OCamlPedia site where
> >
> >
> > The ocamlweb project announced at last year's OCaml User's Meeting can
> serve
> > this purpose. Please see the master repo here [1] and the live site here
> [2]*.
> > We welcome and need the community to add more content. We've attempted
> to make
> > contributing easier. You just checkout a git project, add simple html,
> and
> > submit a pull request. This is perhaps not as nice as a wiki, but it
> let's us
> > scale up to other features that wikis are not suitable for.
> >
> > If you need help getting a larger section started, such as on the topic
> of this
> > thread, just email me or the mailing list [3].
> >
> > [1] https://github.com/agarwal/ocamlweb
> > [2] http://www.ocaml-lang.org/
> > [3] https://lists.forge.ocamlcore.org/cgi-bin/listinfo/ocamlweb-devel
> > * Note the live site is behind the master repo as we have some cleaning
> up yet
> > to do in the master branch.
> >
>
> --
> --Roberto Di Cosmo
>
> ------------------------------------------------------------------
> Professeur               En delegation a l'INRIA
> PPS                      E-mail: roberto@dicosmo.org
> Universite Paris Diderot WWW  : http://www.dicosmo.org
> Case 7014                Tel  : ++33-(0)1-57 27 92 20
> 5, Rue Thomas Mann
> F-75205 Paris Cedex 13   Identica: http://identi.ca/rdicosmo
> FRANCE.                  Twitter: http://twitter.com/rdicosmo
> ------------------------------------------------------------------
> Attachments:
> MIME accepted, Word deprecated
>      http://www.gnu.org/philosophy/no-word-attachments.html
> ------------------------------------------------------------------
> Office location:
>
> Bureau 6C08 (6th floor)
> 175, rue du Chevaleret, XIII
> Metro Chevaleret, ligne 6
> ------------------------------------------------------------------
>

[-- Attachment #2: Type: text/html, Size: 3754 bytes --]

Re: [Caml-list] Distributed computing libraries

[jean-marc alliot]

Le 06/06/2012 18:53, Thomas Braibant a écrit :
> Hi list,
>
> There exists several different "distributed computing libraries" in OCaml:
>
> - Functory http://functory.lri.fr/
> - JoCaml http://jocaml.inria.fr/ (not really a library, though)
> - Nproc https://github.com/MyLifeLabs/nproc
> - OCaml for multicore http://www.algo-prog.info/ocmc/web/ (not a library either)
> - Parmap https://gitorious.org/parmap/parmap
> - Plasma Map/Reduce http://plasma.camlcity.org/plasma/
>
> And I am pretty sure that this list is not exhaustive, looking at
> http://caml.inria.fr/cgi-bin/hump.en.cgi?sort=0&browse=77
>
> Maybe it would be interesting for the community if someone could sum
> up the pros and cons of each of this "libraries", and maybe give some
> information about their status (still in development, mature, etc)?
>
> With best regards,
> Thomas Braibant
>
I was one of the two authors of the ocamlpvm interface. It works well 
and has been used for large "industrial" projects without any particular 
problems. These bindings are especially interesting when you need simple 
communications between different processes, and you also need to be able 
to launch processes by hand. The PVM bindings are stable and robust, PVM 
works for multi-core and  network parallelism but PVM is also slowly dying.

We have also been using lately the MPI bindings written by Xavier Leroy. 
The Ocaml bindings+MPI are an extremely efficient way to write parallel 
applications in the MPI paradigm. We compared it to a "home-made" shared 
memory system for exchanging objects between ocaml processes, and MPI is 
much faster (we were using MPICH2 MPI implementation), with, as a bonus, 
all the MPI goodies (such as transparent network parallelism). Well, 
that's a lesson to learn: leave parallelism implementation to the 
professionals.
We had to add one or two functions to Xavier's bindings to do the job 
(the MPI bindings are not complete, and if I remember correctly the 
iprobe function was not implemented). The modification is trivial.
If there are people interested, I could probably dig up the code.

Jean-Marc