forking, threads and events

[Erik de Castro Lopo <mle+ocaml@mega-nerd.com>]

Hi all,

I'm thinking about to start a new project which has some rather 
critical requirements. It will be compiled with the native compiler 
initially targeting Linux/Unix but eventually also windoze.

The app consists of a main engine which spawns many short lived 
child threads or processes. The children go away, do their work and 
then pass their results back to the main engine. Many of the children 
will spawn another process and read the child process output via a 
pipe and many of the children will block on I/O. A small portion of 
the children may be I/O bound, but there is not way of telling which 
beforehand.

Since I would like to maximize the throughput on multi-core and 
multi-processor machines I am thinking of using a mix of forking and 
threading. For communications, I was thinking of using the Event
module for communication between threads, but I don't think that
works for forked process (pipes maybe?).

Anybody have any advice for this project? Any war stories from similar
projects? Any readings they can recommend?

Thanks,
Erik
-- 
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
"Data is not information, Information is not knowledge, Knowledge is
not understanding, Understanding is not wisdom."
-- Clifford Stoll