From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by yquem.inria.fr (Postfix) with ESMTP id C3DA7BC57 for ; Tue, 30 Nov 2010 17:07:34 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AnABAMCv9EzU4367k2dsb2JhbACDUJ9CFQEBAQEJCQoJEQMfsnWRJAKBH4MzcwSHEIZh X-IronPort-AV: E=Sophos;i="4.59,280,1288566000"; d="scan'208";a="90421623" Received: from moutng.kundenserver.de ([212.227.126.187]) by mail1-smtp-roc.national.inria.fr with ESMTP; 30 Nov 2010 17:07:34 +0100 Received: from office1.lan.sumadev.de (dslb-094-219-222-142.pools.arcor-ip.net [94.219.222.142]) by mrelayeu.kundenserver.de (node=mreu1) with ESMTP (Nemesis) id 0MMrlb-1PKT8c3bVX-0081Sf; Tue, 30 Nov 2010 17:07:34 +0100 Received: from [192.168.1.111] (84-107-230-64.dsl.quicknet.nl [84.107.230.64]) by office1.lan.sumadev.de (Postfix) with ESMTPA id 7B7E65F702; Tue, 30 Nov 2010 17:07:33 +0100 (CET) Subject: Re: Threading and SharedMem (Re: [Caml-list] Re: Is OCaml fast?) From: Gerd Stolpmann To: Stephan Houben Cc: caml-list@yquem.inria.fr In-Reply-To: <4CF518A6.9000602@planet.nl> References: <4CF518A6.9000602@planet.nl> Content-Type: text/plain; charset="UTF-8" Date: Tue, 30 Nov 2010 17:07:31 +0100 Message-ID: <1291133251.16005.604.camel@thinkpad> Mime-Version: 1.0 X-Mailer: Evolution 2.28.1 Content-Transfer-Encoding: 7bit X-Provags-ID: V02:K0:HQhMJQq7V+w4ZT+rjDZ1nryvopB/5T0HkeChNENuum2 dj8kNept6I0x4l/gZgzodsQMAKASC2wbZfR6X2rFD7TVy8yQGQ 32b050Gg9dRysbAVO5tVdYkffwLSS4uVJzrE8iwlpaqSWPcey9 uAxK3x4Xhzy1p5Ogmi17Gcu4qeBvhtdLfYcldUieQO4RCv1VfE ZebYnfD7pb8BTaEH48+2Q== X-Spam: no; 0.00; threading:01 ocaml:01 gerd:01 stolpmann:01 gerd:01 stolpmann:01 model:01 subprocess:01 subprocess:01 descriptors:01 descriptors:01 indirection:01 semaphores:01 real-world:01 darmstadt:01 Am Dienstag, den 30.11.2010, 16:30 +0100 schrieb Stephan Houben: > On 11/30/2010 02:22 PM, Gerd Stolpmann wrote: > > I don't think this is the reason. Many people can ignore Windows, > > actually. > > > > The problem is more that your whole program needs then to be > > restructured - multi-processing implies a process model (which is the > > master, which are the workers). With multi-threading you can start > > threads at all times without having to worry about that (i.e. supports > > "programming without design" if you want to take that as a negative > > point). > > > > This is what I want to fix with my Netmulticore library - it defines a > > framework allowing you to start new processes at any time without having > > to worry about the process hierarchy. > > I have in fact read with much interest your blog at > http://blog.camlcity.org/blog/parallelmm.html . > > Your approach there is to really have separate programs for > server and client. However, one nice thing about fork is that you don't > have to restructure your program; you can just call fork down somewhere > in some subroutine where you decide it is convenient, start doing some > multicore computation, finish and return, and the caller needs never know > that you did that. So you can indeed program without design using fork. Well, I would not recommend that in all cases: fork duplicates all memory, and if this is a lot, you can end up consuming a lot of RAM. Even worse, the GC of the forked subprocess has to manage all of the RAM, including the part that is not required for doing the computation in the subprocess (the copy-on-write optimization of the OS gets you nothing here). Also, there can be subtle interactions between the parent and the child, e.g. file descriptors are inherited, affecting whether closed descriptors can be recognized. So, use with care, and not without design. Forking in the middle of a bigger program can be quite disastrous. > Of course, the advantage of your approach is that you can now distribute > the work over multiple machines. So I guess there is an appropriate > place for all of these techniques. I was recently working on an improved fork machinery, with some indirection between the request for creating a new worker process, and the actual fork. That's this netmulticore library I'm talking about. The new process is not a child of the process requesting the creation, but always a child of a common master process. This avoids all the problems (memory issues, file descriptor issues, and a few more), at the cost of having to transmit state to the new process. > > Also, many practical problems are only O(n log n), at most. The cost for > > serialization of data through a pipe cannot be neglected here. This > > makes shared memory attractive, even if it is only available in a > > restricted form (like write once memory). > > Well, the original context was one of a benchmark which had an > arbitrary rule that you can only use functions from the bundled libraries. > And my proposal was to use the pipe for synchronisation and the shared memory > for bulk communication. > > If we drop the arbitrary rule there are faster options than pipes. > (e.g. POSIX semaphores in a shared memory segment). Yes, it's really arbitrary. All remaining solutions are very restricted then, and don't have much to do with what you would choose for solving a real-world problem. This makes this benchmark irrelevant. Gerd -- ------------------------------------------------------------ Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany gerd@gerd-stolpmann.de http://www.gerd-stolpmann.de Phone: +49-6151-153855 Fax: +49-6151-997714 ------------------------------------------------------------