From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 518BF7EE80 for ; Wed, 20 Mar 2013 10:32:59 +0100 (CET) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of rdicosmo@gmail.com) identity=pra; client-ip=209.85.212.175; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="rdicosmo@gmail.com"; x-sender="rdicosmo@gmail.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of rdicosmo@gmail.com designates 209.85.212.175 as permitted sender) identity=mailfrom; client-ip=209.85.212.175; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="rdicosmo@gmail.com"; x-sender="rdicosmo@gmail.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mail-wi0-f175.google.com) identity=helo; client-ip=209.85.212.175; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="rdicosmo@gmail.com"; x-sender="postmaster@mail-wi0-f175.google.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AtcJALeASVHRVdSviGdsb2JhbABDhTiCWqpejHGFLAEDgUkWDgEBARMUFAQkgiQBAQMCIw8BDQEUGQsBAwwBBQUYAgIFBB0CAg8FIAEFAQEJGBMJCQmHZwMPBAMFo1qLY4NKhQ8nAwqJVQEFDAmBDowhFQYaOYMRMmEDll+BIIpMgzU/gVmCVYFpCRc X-IPAS-Result: AtcJALeASVHRVdSviGdsb2JhbABDhTiCWqpejHGFLAEDgUkWDgEBARMUFAQkgiQBAQMCIw8BDQEUGQsBAwwBBQUYAgIFBB0CAg8FIAEFAQEJGBMJCQmHZwMPBAMFo1qLY4NKhQ8nAwqJVQEFDAmBDowhFQYaOYMRMmEDll+BIIpMgzU/gVmCVYFpCRc X-IronPort-AV: E=Sophos;i="4.84,876,1355094000"; d="scan'208";a="8447120" Received: from mail-wi0-f175.google.com ([209.85.212.175]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/RC4-SHA; 20 Mar 2013 10:32:58 +0100 Received: by mail-wi0-f175.google.com with SMTP id l13so4987463wie.2 for ; Wed, 20 Mar 2013 02:32:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=LLFCBLmXdufI0oJPQhXDm5YCw1V9LFkfWyHKq85AOWY=; b=hGPZNIWOIpmTrsf3r4EysBWEp8gdbQ88oiVyNehBOjs5gKOzw3p65HmelRab0uliSF lsiFsdm7VEzYuGpmM618kRrGTnTHL2zKJbIyVRSxaqAyqwYKaIrnZfCoTxBJHMMAH1x+ 0BMeZrQhvWTYqG4BdI3sfNCG7Lii13pvCqvaf1/Ws+FI5Qo4vW4GbiaHttYD4SyB/xeM APItYnomh3qaTx6mR7MGsWGmKA0VdNsRmW5RJRAKvznNYsKDIqZwidev5vAaa9lboejc 1ZgIOb5uQ4a5p9ETXxtSXTRDC158itMf1wayEpGIm/ycDzux990bt/Cb/Ner7igW1S63 PI7w== X-Received: by 10.194.22.5 with SMTP id z5mr9107594wje.5.1363771978312; Wed, 20 Mar 2013 02:32:58 -0700 (PDT) Received: from voyager (bny92-3-81-56-44-163.fbx.proxad.net. [81.56.44.163]) by mx.google.com with ESMTPS id j4sm5664744wiz.10.2013.03.20.02.32.56 (version=TLSv1 cipher=RC4-SHA bits=128/128); Wed, 20 Mar 2013 02:32:57 -0700 (PDT) Sender: Roberto Di Cosmo Received: from dicosmo by voyager with local (Exim 4.72) (envelope-from ) id 1UIFOB-0001gl-Kj; Wed, 20 Mar 2013 10:32:55 +0100 Date: Wed, 20 Mar 2013 10:32:55 +0100 From: Roberto Di Cosmo To: Jean-Marc Alliot Cc: caml-list@inria.fr, Marco Danelutto Message-ID: <20130320093255.GB6013@voyager> References: <00ba01ce200c$d23f1910$76bd4b30$@ffconsultancy.com> <010801ce201f$afd6ad80$0f840880$@ffconsultancy.com> <037b01ce2307$d54e6130$7feb2390$@ffconsultancy.com> <51485E77.9010502@irit.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <51485E77.9010502@irit.fr> User-Agent: Mutt/1.5.20 (2009-06-14) Subject: Re: [Caml-list] Case study in optimization: porting a compiler from OCaml to F# Dear Jean-Marc, we are quite happy that Parmap is useful in your applications, and would love to know more about your use cases. Btw, I uploaded to HAL an author version of the article published in Procedia Computer Science, so that everybody can access the paper ... it is available at http://hal.archives-ouvertes.fr/docs/00/69/25/15/PDF/parmap-author-file.pdf all the best -- Roberto and Marco On Tue, Mar 19, 2013 at 01:47:51PM +0100, Jean-Marc Alliot wrote: > We have been using Parmap and are quite happy with it (we use it as > an alternative to the Ocaml/MPI implementation when MPI is not > strictly required), with excellent scalability on our applications > (even if we had to change from time to time a little bit or our > code, regarding the fact that Parmap does not allow "in place" > modifications). > > Le 17/03/2013 13:06, Jon Harrop a écrit : > >What happens if the inner function returns results via mutation? I > >assume you must rearrange the code to return all results > >explicitly and they will then be deep copied (which destroys > >scalability due to limited shared memory bandwidth on multicores). > As fas as I can tell there is no "copy" of any form. Parmap only > collects results. If you do "in place" modifications, they are lost. > Parmap is, in a way, "functional"... > >Does it do load balancing? I assume not given that ncores is > >hardcoded. > There is an optional argument (chunksize) to manually control load > balancing. > >Does a parmap with ncores=4 inside a parmap with ncores=4 create > >16 processes? Does it deep copy inputs and/or outputs? > Regarding outputs, Parmap uses a shared memory area and > Marshal/Unmarshal to collect outputs. There is an optimization done > when array of floats are returned, as marshalling is not used, thus > reducing the overhead. > >I assume so, at least for outputs, because you cannot write > >results in-place without a shared mutable heap. Does parmap have a > >large constant overhead? I assume so if it is forking processes. > Well, it depends on what you call "large constant overhead". Forking > is not a so expensive primitive on modern Unix systems, because > pages are only copied when written-to (copy-on-write). > >Another solution is to prefork and explicitly communicate all > >inputs using message passing but this is equally problematic. You > >have to rearrange the code. Deep copying inputs also destroys > >scalability. Cheers, Jon. > > There is an article describing the implementation (I think it is > also available online): > > A “minimal disruption” skeleton experiment: seamless map & reduce > embedding in OCaml > M. Danelutto, R. Di Cosmo > Procedia Computer Science 9 ( 2012 ) 1837 – 1846 > > > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs -- Roberto Di Cosmo ------------------------------------------------------------------ Professeur En delegation a l'INRIA PPS E-mail: roberto@dicosmo.org Universite Paris Diderot WWW : http://www.dicosmo.org Case 7014 Tel : ++33-(0)1-57 27 92 20 5, Rue Thomas Mann F-75205 Paris Cedex 13 Identica: http://identi.ca/rdicosmo FRANCE. Twitter: http://twitter.com/rdicosmo ------------------------------------------------------------------ Attachments: MIME accepted, Word deprecated http://www.gnu.org/philosophy/no-word-attachments.html ------------------------------------------------------------------ Office location: Bureau 320 (3rd floor) Batiment Sophie Germain Avenue de France Metro Bibliotheque Francois Mitterrand, ligne 14/RER C ----------------------------------------------------------------- GPG fingerprint 2931 20CE 3A5A 5390 98EC 8BFC FCCA C3BE 39CB 12D3