From mboxrd@z Thu Jan 1 00:00:00 1970 X-Sympa-To: caml-list@inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id p3M8Y9IS024240 for ; Fri, 22 Apr 2011 10:34:09 +0200 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AokDABk9sU3B/BfUcGdsb2JhbAAwhCChIgEKCg4HFAMisXoBkQSBJoNQgQAEji2EB4lg X-IronPort-AV: E=Sophos;i="4.64,253,1301868000"; d="scan'208";a="97742332" Received: from msa03.smtpout.orange.fr (HELO msa.smtpout.orange.fr) ([193.252.23.212]) by mail2-smtp-roc.national.inria.fr with ESMTP; 22 Apr 2011 10:34:04 +0200 Received: from [192.168.1.63] ([83.204.179.232]) by mwinf5d12 with ME id aYa31g00F51EXVP03Ya3AQ; Fri, 22 Apr 2011 10:34:04 +0200 Message-ID: <4DB13D7B.50900@lexifi.com> Date: Fri, 22 Apr 2011 10:34:03 +0200 From: Alain Frisch Organization: LexiFi User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Christophe TROESTLER CC: caml-list@inria.fr References: <76544177.594058.1303341821437.JavaMail.root@zmbs4.inria.fr> <4DAFE141.7080003@inria.fr> <4DAFF442.8000806@lexifi.com> <20110421.210304.1267840107736400776.Christophe.Troestler+ocaml@umons.ac.be> In-Reply-To: <20110421.210304.1267840107736400776.Christophe.Troestler+ocaml@umons.ac.be> Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit X-Validation-by: alain@frisch.fr Subject: Re: [Caml-list] Efficient OCaml multicore -- roadmap? On 04/21/2011 09:03 PM, Christophe TROESTLER wrote: > On Thu, 21 Apr 2011 11:09:22 +0200, Alain Frisch wrote: >> >> On 04/21/2011 09:48 AM, Fabrice Le Fessant wrote: >>> You think the programmers in the world that are only interested in >>> floating-point intensive computations, with fine-grain >>> concurrency, are a huge majority. I think they are not so >>> many. Can we do a better job of quantifying this ? >> >> [...] I guess we are in the nice niche of "embarassingly parallel" >> tasks. Again, not so much because of the structure of the numerical >> problem itself, but because we need to run many independent >> instances of the problem. > > May you say why Functory is not good enough for this ? > http://www.lri.fr/~filliatr/functory/About.html I did not say Functory is not good enough for this. Actually, I did not know about this project. Although my message was not explicit about it, we have implemented our own solution, based on a multi-process approach, and we are quite happy with it. That said, we follow with a lot of interest all the projects and discussions about making it easier to write distributed computations in OCaml. After quickly reading the API and the technical report, I think the library would indeed be a good match for the kind of computations we are dealing with. Actually, it has a lot in common with the solution we have implemented. Some difference between Functory and our solution: - Functory's Cores module relies on system calls which are not available under Windows. Our solution is based on a different technique: external worker processed are spawned by the main program by executing argv[0] with a special command-line argument (to distinguish from a regular invocation). Our "Computations" library intercepts this argument and runs a loop that waits for new tasks (similar to Functory's Network solutions). - Functory's external worker processes execute a single task and die. In our solution, we maintain a pool of available worker processes; when a worker process has finished computing a task, it notifies the scheduler explicitly. This avoids some non-negligible start-up costs (especially under Windows) and allows workers to keep some in-memory caches and database connections. Again, this looks similar to Functory's Network approach. - Our API allows workers to notify the caller with intermediate results and progress feedback (typically, to display some progress bar and status to the user). For instance, a single Monte Carlo pricing task reports progress percentage in the GUI, together with the current price estimate and observed variance. Similarly, a "portfolio" pricing task reports the percentage (number of contracts already priced). The API also allows the caller to kill a running task (typically when the user clicks on "Stop" in the GUI). We do this by asking the OS to kill the worker process. The caller can still use intermediate results to do something useful. - We also have some solution for distributing computations across the network (not in use in production yet), but we go through our application's database instead of direct TCP networking. Any connected client makes its cores available to take some tasks from the task pool; and it can itself submit new tasks. (This also give a cache to share the result of computations between different clients.) Cheers, Alain