From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 742C87EE99 for ; Wed, 8 Jan 2014 02:12:44 +0100 (CET) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of berenger@riken.jp) identity=pra; client-ip=134.160.33.161; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="berenger@riken.jp"; x-sender="berenger@riken.jp"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of berenger@riken.jp designates 134.160.33.161 as permitted sender) identity=mailfrom; client-ip=134.160.33.161; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="berenger@riken.jp"; x-sender="berenger@riken.jp"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of postmaster@postman.riken.jp designates 134.160.33.161 as permitted sender) identity=helo; client-ip=134.160.33.161; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="berenger@riken.jp"; x-sender="postmaster@postman.riken.jp"; x-conformance=sidf_compatible; x-record-type="v=spf1" X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AsQBAPakzFKGoCGhnGdsb2JhbABZg0O6RYElDgEBAQEBCAsJCRQogiUBAQEDATg1CwYLCxgJFg8JAwIBAgFFBg0IAQGHeAgNxBmPC4Q3AQOJQY5WgTCFFY8M X-IPAS-Result: AsQBAPakzFKGoCGhnGdsb2JhbABZg0O6RYElDgEBAQEBCAsJCRQogiUBAQEDATg1CwYLCxgJFg8JAwIBAgFFBg0IAQGHeAgNxBmPC4Q3AQOJQY5WgTCFFY8M X-IronPort-AV: E=Sophos;i="4.95,621,1384297200"; d="scan'208";a="52169894" Received: from postman1.riken.jp (HELO postman.riken.jp) ([134.160.33.161]) by mail2-smtp-roc.national.inria.fr with ESMTP; 08 Jan 2014 02:12:42 +0100 Received: from postman.riken.jp (postman1.riken.jp [127.0.0.1]) by postman.riken.jp (Postfix) with SMTP id 24A0932C03B7 for ; Wed, 8 Jan 2014 10:12:39 +0900 (JST) Received: from [172.27.98.103] (rikad98.riken.jp [134.160.214.98]) by postman.riken.jp (Postfix) with ESMTPA id 492A432A0090 for ; Wed, 8 Jan 2014 10:12:38 +0900 (JST) Message-ID: <52CCA606.4050209@riken.jp> Date: Wed, 08 Jan 2014 10:12:38 +0900 From: Francois Berenger User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: caml-list References: <20140107200328.GA14297@voyager> In-Reply-To: <20140107200328.GA14297@voyager> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-PMX-Version: 6.0.0.2142326, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2014.1.8.10615 Subject: Re: [Caml-list] Concurrent/parallel programming On 01/08/2014 05:03 AM, Roberto Di Cosmo wrote: > Dear Yotam, > speeding up a computation by parallelising part of it, as you are probably > finding out the hard way, is not an easy task. > > Being the main developer of Parmap, I am quite curious to know where the > "complete failure" you are mentioning actually comes from, and I would like > to put you in touch with Francois Berenger, that has been using Parmap in > production for quite a while with excellent results. Absolutely! I usually develop a sequential program working on a big list and using List.iter or List.map. When I am happy with the sequential program, I add an nprocs parameter to use the Parmap equivalents when nprocs > 1. The only problems I have encountered were out of memory in case the nprocs forks did not fit on my machine when I handled too big data. I usually handle independent pieces of data, for example many proteins or chemical molecules. More answers follow below. > All the best > > -- > Roberto > > > On Tue, Jan 07, 2014 at 02:54:33PM -0500, Yotam Barnoy wrote: >> Hi List >> >> So far, I've been programming in ocaml using only sequential programs. In my >> last project, which was an implementation of a large machine learning >> algorithm, I tried to speed up computation using a little bit of parallelism >> with ParMap, and it was a complete failure. It's possible that more time would >> have yielded better results, but I just didn't have the time to invest in it >> given how bad the initial results were. >> >> My question is, what are the options right now as far as parallelism is >> concerned? I'm not talking about cooperative multitasking, but about really >> taking advantage of multiple cores. I'm well aware of the runtime lock and I'm >> ok with message passing between processes or a shared area in memory, but I'd >> rather have something more high level than starting up several processes, >> creating a named pipe or a socket, and trying to pass messages through that. There are MPI bindings for OCaml. >> Also, I assume that using a shared area in memory involves some C code? Am I >> wrong about that? Gerd Stolpman's impressive ocamlnet library has nice pure OCaml wrappers for all this I believe. http://projects.camlcity.org/projects/ocamlnet.html >> I was expecting Core's Async to fill this role, but realworldocaml is fuzzy on >> this topic, apparently preferring to dwell on cooperative multitasking (which >> is fine but not what I'm looking for), and I couldn't find any other >> documentation that was clearer. Cf. parallel Vs. concurrent in there: http://chimera.labs.oreilly.com/books/1230000000929/ch01.html#sec_terminology I believe Parmap and MPI are for parallelism (what you want very probably), while Async/Lwt and OCaml threads are for concurrency (used in webservers, GUIs, etc.). -- Best regards, Francois Berenger.