forking, threads and events

forking, threads and events

[Erik de Castro Lopo]

Hi all,

I'm thinking about to start a new project which has some rather 
critical requirements. It will be compiled with the native compiler 
initially targeting Linux/Unix but eventually also windoze.

The app consists of a main engine which spawns many short lived 
child threads or processes. The children go away, do their work and 
then pass their results back to the main engine. Many of the children 
will spawn another process and read the child process output via a 
pipe and many of the children will block on I/O. A small portion of 
the children may be I/O bound, but there is not way of telling which 
beforehand.

Since I would like to maximize the throughput on multi-core and 
multi-processor machines I am thinking of using a mix of forking and 
threading. For communications, I was thinking of using the Event
module for communication between threads, but I don't think that
works for forked process (pipes maybe?).

Anybody have any advice for this project? Any war stories from similar
projects? Any readings they can recommend?

Thanks,
Erik
-- 
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
"Data is not information, Information is not knowledge, Knowledge is
not understanding, Understanding is not wisdom."
-- Clifford Stoll

Re: [Caml-list] forking, threads and events

[Gerd Stolpmann]

Am Dienstag, den 10.10.2006, 22:16 +1000 schrieb Erik de Castro Lopo:
> Hi all,
> 
> I'm thinking about to start a new project which has some rather 
> critical requirements. It will be compiled with the native compiler 
> initially targeting Linux/Unix but eventually also windoze.
> 
> The app consists of a main engine which spawns many short lived 
> child threads or processes. The children go away, do their work and 
> then pass their results back to the main engine. Many of the children 
> will spawn another process and read the child process output via a 
> pipe and many of the children will block on I/O. A small portion of 
> the children may be I/O bound, but there is not way of telling which 
> beforehand.
> 
> Since I would like to maximize the throughput on multi-core and 
> multi-processor machines I am thinking of using a mix of forking and 
> threading.

You know that there is no fork on Windows?

>  For communications, I was thinking of using the Event
> module for communication between threads, but I don't think that
> works for forked process (pipes maybe?).
> 
> Anybody have any advice for this project? Any war stories from similar
> projects? Any readings they can recommend?

If there wasn't the Windows requirement I could recommend this:

You may have a look at ocamlnet2. It includes the netplex library that
manages parent/children relationships between either forked processes or
threads. You can use SunRPC for communication (or whatever you like, but
SunRPC support is included). The bad news is that it does not run on the
native Windows port, not even in the threaded variant (because there's
no socketpair...).

Ah yes, I have a war story, but cannot tell it now. I can only say it is
used for a really big commercial project.

Download it here:

http://www.ocaml-programming.de/packages/ocamlnet-2.2test13.tar.gz

An online manual is here:

http://ocamlnet.sourceforge.net/manual-2.2/

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] forking, threads and events

[Erik de Castro Lopo]

skaller wrote:

> The correct way to do this is one pthread per CPU (called a
> worker thread) and use fibres (synchronous threads) within
> those pthreads. Then you will need a thread for I/O.
> 
> You really should use Felix.

I'm pushing for Ocaml on this project. If that doesn't fit
the bill there are others in the team pushing for Erlang.

> If you don't need performance .. why did you mention using
> multiple cores? One core will do.

I'm trying to optimize performance of lots of small sub-processes
which are likely to block on I/O. I'm therefore trying to start
as many as possible simultaneously rather than run them one by
one serially.

> Felix runs on Win32 using any C++ compiler, including MSVC++
> and uses the native Win32 API. It's currently not only
> the logical choice for high performance high level asynchronous
> programming .. its also the ONLY choice

I have been led to believe that Erlang also fits the bill.

> unless you count C++ as 'high level' :)

I call C++ a curse on programmers everywhere; the language
that has enabled and encouraged more stupidity and bad software 
design than any other programming language ever.

Erik
-- 
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
"It is forbidden for a Muslim to be a loyal friend to someone who does
not believe in God and His Prophet, or someone who fights the religion 
of Islam." -- Fifth grade text book from Saudi Arabia
http://www.washingtonpost.com/wp-dyn/content/article/2006/05/19/AR2006051901769_pf.html

Re: [Caml-list] forking, threads and events

[Erik de Castro Lopo]

Gerd Stolpmann wrote:

> You know that there is no fork on Windows?

Yeah, but I was thinking of using the Cygwin version which I 
think does have fork (and select which is also missing).

> If there wasn't the Windows requirement I could recommend this:
> 
> You may have a look at ocamlnet2. It includes the netplex library that
> manages parent/children relationships between either forked processes or
> threads. You can use SunRPC for communication (or whatever you like, but
> SunRPC support is included). The bad news is that it does not run on the
> native Windows port, not even in the threaded variant (because there's
> no socketpair...).

I hate windows. There are so many good, reliable, well designed
OSes out there and people on those good OSes still have to work
around the fact the the vast majority of people are on a broken
piece of crap like windows.

Erik
-- 
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
The main confusion about C++ is that its practitioners think
it is simultaneously a  high and low level language when in
reality it is good at neither.