From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id pAULQtup009424 for ; Wed, 30 Nov 2011 22:26:55 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkUBAJye1k5CbwQbkWdsb2JhbABEFqp8IgEBAQEJCwsbBCGBcgEBBAFAOAEBBAsLGAkWDwkDAgECAUUGDQEHAQGIAwimDZI9B4sgiCyRcoxu X-IronPort-AV: E=Sophos;i="4.71,273,1320620400"; d="scan'208";a="121570543" Received: from out3.smtp.messagingengine.com ([66.111.4.27]) by mail4-smtp-sop.national.inria.fr with ESMTP/TLS/ADH-AES256-SHA; 30 Nov 2011 22:26:49 +0100 Received: from compute6.internal (compute6.nyi.mail.srv.osa [10.202.2.46]) by gateway1.nyi.mail.srv.osa (Postfix) with ESMTP id 133AA21B72 for ; Wed, 30 Nov 2011 16:26:49 -0500 (EST) Received: from frontend2.nyi.mail.srv.osa ([10.202.2.161]) by compute6.internal (MEProxy); Wed, 30 Nov 2011 16:26:49 -0500 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=message-id:date:from:mime-version:to:cc :subject:references:in-reply-to:content-type :content-transfer-encoding; s=smtpout; bh=2nsZ3AKNjG5rKQdZZX7kN0 2G16A=; b=r9yV4pGaS/Ab9b1OZ0OjJXhQVpUgCPwS09cGMYP9VYGXMwaH0GWjEy m57LPaH4TqPYh6351IQw+Jm9fnGiCqknJSVlyAk6w1FJY97tH6dLwG+qvKhI/1xh VMN62IXQ0iElFs8H0X49WArUKPa1do7NK5mCOVnWh6F74/k3zB5VE= X-Sasl-enc: OXn+tsetcz1ejBmXsb8hYCRMlphIcyITcvH8tbl245DO 1322688408 Received: from [192.168.1.206] (64-71-1-165.static.wiline.com [64.71.1.165]) by mail.messagingengine.com (Postfix) with ESMTPSA id A883B48247B; Wed, 30 Nov 2011 16:26:48 -0500 (EST) Message-ID: <4ED6A102.9090601@ens-lyon.org> Date: Wed, 30 Nov 2011 13:32:50 -0800 From: Martin Jambon User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.9) Gecko/20101023 Lightning/1.0b3pre Thunderbird/3.1.3 MIME-Version: 1.0 To: Jerome Vouillon CC: OCaml Mailing List References: <4ED59AF6.1030506@ens-lyon.org> <20111130093557.GA19471@pps.jussieu.fr> In-Reply-To: <20111130093557.GA19471@pps.jussieu.fr> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Caml-list] Nproc: process pools for OCaml (request for suggestions) On 11/30/2011 01:35 AM, Jerome Vouillon wrote: > Hi, > > On Tue, Nov 29, 2011 at 06:54:46PM -0800, Martin Jambon wrote: >> I would like to publicize Nproc, which is an implementation of process >> pools for OCaml based on fork, pipes, Marshal and Lwt: >> >> https://github.com/MyLifeLabs/nproc >> >> Using Nproc involves: >> >> 1. Creating a pool of N processes, N being chosen by the user. >> 2. Running tasks: >> a. Submitting a task (f, x) of any type. >> b. Defining what to do when the result becomes available. > > Marco Danelutto and Roberto Di Cosmo have written a small library to > perform parallel maps and folds on multi-core machines. This is > complementary to your library. You should look at the implementation: > communication is performed by marshalling to a shared memory area for > better performances, and pipes are used only for synchronization. > > https://gitorious.org/parmap/ > > Thank you, this is interesting. I hadn't look at the implementation although I knew about parmap but wanted something with an Lwt-ready interface. I also wanted something that could handle continuous streams, without having to create a process for each stream item (assuming fork() is more expensive than we want). > I have also written a more low-level library, where you can control > which process runs each task. This is useful when the processes have > to work with a lot of data that you don't want to duplicate. You can > get the code with the following command: > > darcs clone http://www.pps.jussieu.fr/~vouillon/coinst/darcs/dev/ > > The file of interest is task.ml. I see. Thanks. Martin