From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=none autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by yquem.inria.fr (Postfix) with ESMTP id 17033BBE1 for ; Tue, 10 Oct 2006 14:16:15 +0200 (CEST) Received: from smtp.syd.people.net.au (smtp.syd.people.net.au [218.214.225.98]) by concorde.inria.fr (8.13.6/8.13.6) with SMTP id k9ACGAET007663 for ; Tue, 10 Oct 2006 14:16:13 +0200 Received: (qmail 3723 invoked from network); 10 Oct 2006 12:16:25 -0000 Received: from unknown (HELO coltrane.mega-nerd.net) (218.214.64.136) by smtp.syd.people.net.au with SMTP; 10 Oct 2006 12:16:25 -0000 Received: from coltrane (coltrane [192.168.1.101]) by coltrane.mega-nerd.net (Postfix) with SMTP id 82F4C7B69 for ; Tue, 10 Oct 2006 22:16:07 +1000 (EST) Date: Tue, 10 Oct 2006 22:16:07 +1000 From: Erik de Castro Lopo To: caml-list@inria.fr Subject: forking, threads and events Message-Id: <20061010221607.7ecc5cba.mle+ocaml@mega-nerd.com> Organization: Erik Conspiracy Secret Labs X-Mailer: Sylpheed version 2.2.6 (GTK+ 2.8.18; i486-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Miltered: at concorde with ID 452B8F0B.000 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Spam: no; 0.00; ocaml:01 threads:01 compiler:01 unix:01 threads:01 threading:01 short:01 data:02 pipes:02 native:02 output:02 module:03 erik:04 erik:04 compiled:04 Hi all, I'm thinking about to start a new project which has some rather critical requirements. It will be compiled with the native compiler initially targeting Linux/Unix but eventually also windoze. The app consists of a main engine which spawns many short lived child threads or processes. The children go away, do their work and then pass their results back to the main engine. Many of the children will spawn another process and read the child process output via a pipe and many of the children will block on I/O. A small portion of the children may be I/O bound, but there is not way of telling which beforehand. Since I would like to maximize the throughput on multi-core and multi-processor machines I am thinking of using a mix of forking and threading. For communications, I was thinking of using the Event module for communication between threads, but I don't think that works for forked process (pipes maybe?). Anybody have any advice for this project? Any war stories from similar projects? Any readings they can recommend? Thanks, Erik -- +-----------------------------------------------------------+ Erik de Castro Lopo +-----------------------------------------------------------+ "Data is not information, Information is not knowledge, Knowledge is not understanding, Understanding is not wisdom." -- Clifford Stoll