Just a comment FWIW, I personally moved away from Parmap because it doesn't deal well with exceptions raised in the child processes. I know the brokenness of exception marshalling (PR#0001961) complicates this, but I came up with something I felt was pretty reasonable in my iteration of the same concept: https://github.com/mlin/forkwork Of course I didn't reimplement some of Parmap's fancier features, but this solution has been "just right" for my applications. I hope this can inspire a future improvement to exception handling in Parmap. Mike On Wed, Mar 20, 2013 at 3:35 PM, Roberto Di Cosmo wrote: > Hi Jon, > a concrete set of well justified benchmarks could serve > the cause more than any abstract discussion; please feel > free to set it up, run it, and analyze the results. > > Having spent quite a bit of energy on Parmap, after it > started as a sort of a one-afternoon project, and with > the experience of the now very old OCamlP3l library that > started much of this at the end of the '90s (including a > detour through an experimental reimplementation in Haskell), > I definitely took the St Thomas stance with this kind of issues :-) > > -- > Roberto > > On Wed, Mar 20, 2013 at 08:54:59PM -0000, Jon Harrop wrote: > > > > Not just the granularity. Also the communication including any > communication involved in scatter and gather phases. That differs a lot > more between OCaml and F#. Fork does copy-on-write but (IIRC) the GC can > incur unnecessary copying but, more importantly, requires the gather phase > to deep copy results back to the original process. In contrast, data can be > passed by reference in F#. > > > > Would be very interesting to benchmark this... > > > > Cheers, > > Jon. > > > > -----Original Message----- > > From: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] On > Behalf Of Francois Berenger > > Sent: 19 March 2013 01:50 > > To: caml-list@inria.fr > > Subject: Re: [Caml-list] Case study in optimization: porting a compiler > from OCaml to F# > > > > I have observed and measured perfect scalability with up to 4 cores of > an OCaml program using Parmap. > > With more than 4 cores, the scalability was degrading. > > > > I think the scalability of the program depends only on the granularity > of the tasks. The tasks were coarse in my case. > > > > F. > > > > On 03/17/2013 09:06 PM, Jon Harrop wrote: > > > Pierre-Alexandre Voye wrote: > > >> So you could maybe use Parmap.map ? > > >> Parmap.parmap ~ncores:4 funct (Parmap.L elem_list) > > > > > > What happens if the inner function returns results via mutation? I > assume you must rearrange the code to return all results explicitly and > they will then be deep copied (which destroys scalability due to limited > shared memory bandwidth on multicores). > > > > > > Does it do load balancing? I assume not given that ncores is hardcoded. > > > > > > Does a parmap with ncores=4 inside a parmap with ncores=4 create 16 > processes? > > > > > > Does it deep copy inputs and/or outputs? I assume so, at least for > outputs, because you cannot write results in-place without a shared mutable > heap. > > > > > > Does parmap have a large constant overhead? I assume so if it is > forking processes. > > > > > > Another solution is to prefork and explicitly communicate all inputs > using message passing but this is equally problematic. You have to > rearrange the code. Deep copying inputs also destroys scalability. > > > > > > Cheers, > > > Jon. > > > > > > > > > > > > > > > -- > > Caml-list mailing list. Subscription management and archives: > > https://sympa.inria.fr/sympa/arc/caml-list > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > > Bug reports: http://caml.inria.fr/bin/caml-bugs > > > > > > -- > > Caml-list mailing list. Subscription management and archives: > > https://sympa.inria.fr/sympa/arc/caml-list > > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > > Bug reports: http://caml.inria.fr/bin/caml-bugs > > -- > Roberto Di Cosmo > > ------------------------------------------------------------------ > Professeur En delegation a l'INRIA > PPS E-mail: roberto@dicosmo.org > Universite Paris Diderot WWW : http://www.dicosmo.org > Case 7014 Tel : ++33-(0)1-57 27 92 20 > 5, Rue Thomas Mann > F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo > FRANCE. Twitter: http://twitter.com/rdicosmo > ------------------------------------------------------------------ > Attachments: > MIME accepted, Word deprecated > http://www.gnu.org/philosophy/no-word-attachments.html > ------------------------------------------------------------------ > Office location: > > Bureau 320 (3rd floor) > Batiment Sophie Germain > Avenue de France > Metro Bibliotheque Francois Mitterrand, ligne 14/RER C > ----------------------------------------------------------------- > GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3 > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs >