From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 67D677EE80 for ; Tue, 19 Mar 2013 13:48:03 +0100 (CET) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jean-marc.alliot@irit.fr) identity=pra; client-ip=147.127.176.11; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jean-marc.alliot@irit.fr"; x-sender="jean-marc.alliot@irit.fr"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of jean-marc.alliot@irit.fr) identity=mailfrom; client-ip=147.127.176.11; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jean-marc.alliot@irit.fr"; x-sender="jean-marc.alliot@irit.fr"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of postmaster@n7smtp.enseeiht.fr designates 147.127.176.11 as permitted sender) identity=helo; client-ip=147.127.176.11; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="jean-marc.alliot@irit.fr"; x-sender="postmaster@n7smtp.enseeiht.fr"; x-conformance=sidf_compatible; x-record-type="v=spf1" X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqECAH9dSFGTf7ALgWdsb2JhbABDiC66B4JugWIWDgEBFiYogiUBBSMPAQVAEQsaAgUWCwICCQMCAQIBRRMIAQEXh3kEr3eCQJAjgSOMIBWBPRaCF4ETA5ZfhX2OEIFp X-IPAS-Result: AqECAH9dSFGTf7ALgWdsb2JhbABDiC66B4JugWIWDgEBFiYogiUBBSMPAQVAEQsaAgUWCwICCQMCAQIBRRMIAQEXh3kEr3eCQJAjgSOMIBWBPRaCF4ETA5ZfhX2OEIFp X-IronPort-AV: E=Sophos;i="4.84,872,1355094000"; d="scan'208";a="8304256" Received: from n7smtp.enseeiht.fr ([147.127.176.11]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 19 Mar 2013 13:48:03 +0100 Received: from imap.enseeiht.fr (imap.enseeiht.fr [147.127.176.21]) by n7smtp.enseeiht.fr (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id r2JCm0CZ018351 for ; Tue, 19 Mar 2013 13:48:02 +0100 Received: from [10.31.1.2] (AToulouse-159-1-73-63.w92-134.abo.wanadoo.fr [92.134.216.63]) (authenticated bits=0) by imap.enseeiht.fr (8.14.3/8.14.3/Debian-5+lenny1) with ESMTP id r2JClvlN016694 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT) for ; Tue, 19 Mar 2013 13:48:00 +0100 Message-ID: <51485E77.9010502@irit.fr> Date: Tue, 19 Mar 2013 13:47:51 +0100 From: Jean-Marc Alliot User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130307 Thunderbird/17.0.4 MIME-Version: 1.0 To: caml-list@inria.fr References: <00ba01ce200c$d23f1910$76bd4b30$@ffconsultancy.com> <010801ce201f$afd6ad80$0f840880$@ffconsultancy.com> <037b01ce2307$d54e6130$7feb2390$@ffconsultancy.com> In-Reply-To: <037b01ce2307$d54e6130$7feb2390$@ffconsultancy.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-3.0 (n7smtp.enseeiht.fr [147.127.176.11]); Tue, 19 Mar 2013 13:48:02 +0100 (CET) X-Scanned-By: MIMEDefang 2.64 on 147.127.176.11 Subject: Re: [Caml-list] Case study in optimization: porting a compiler from OCaml to F# We have been using Parmap and are quite happy with it (we use it as an alternative to the Ocaml/MPI implementation when MPI is not strictly required), with excellent scalability on our applications (even if we had to change from time to time a little bit or our code, regarding the fact that Parmap does not allow "in place" modifications). Le 17/03/2013 13:06, Jon Harrop a écrit : > What happens if the inner function returns results via mutation? I > assume you must rearrange the code to return all results explicitly > and they will then be deep copied (which destroys scalability due to > limited shared memory bandwidth on multicores). As fas as I can tell there is no "copy" of any form. Parmap only collects results. If you do "in place" modifications, they are lost. Parmap is, in a way, "functional"... > Does it do load balancing? I assume not given that ncores is hardcoded. There is an optional argument (chunksize) to manually control load balancing. > Does a parmap with ncores=4 inside a parmap with ncores=4 create 16 > processes? Does it deep copy inputs and/or outputs? Regarding outputs, Parmap uses a shared memory area and Marshal/Unmarshal to collect outputs. There is an optimization done when array of floats are returned, as marshalling is not used, thus reducing the overhead. > I assume so, at least for outputs, because you cannot write results > in-place without a shared mutable heap. Does parmap have a large > constant overhead? I assume so if it is forking processes. Well, it depends on what you call "large constant overhead". Forking is not a so expensive primitive on modern Unix systems, because pages are only copied when written-to (copy-on-write). > Another solution is to prefork and explicitly communicate all inputs > using message passing but this is equally problematic. You have to > rearrange the code. Deep copying inputs also destroys scalability. > Cheers, Jon. There is an article describing the implementation (I think it is also available online): A “minimal disruption” skeleton experiment: seamless map & reduce embedding in OCaml M. Danelutto, R. Di Cosmo Procedia Computer Science 9 ( 2012 ) 1837 – 1846