Fwd: [Caml-list] Faking concurrency using Unix forks and pipes

Fwd: [Caml-list] Faking concurrency using Unix forks and pipes

[Jonathan Bryant]

Oops.  Forgot to CC the list...

Begin forwarded message:

> From: Jonathan Bryant <jtbryant@valdosta.edu>
> Date: May 30, 2007 3:43:50 PM EDT
> To: Erik de Castro Lopo <mle+ocaml@mega-nerd.com>
> Subject: Re: [Caml-list] Faking concurrency using Unix forks and pipes
>
>
> On May 30, 2007, at 3:14 PM, Erik de Castro Lopo wrote:
>
>>
>> To exploit multi-process message-passing style concurrency you  
>> need to
>> fork early before much has been allocated.
>
> What I've always done to get around this is fork a process at the  
> very beginning of the code, and whenever I need to fork a process  
> later, I fork one from that process.  That way almost nothing is in  
> the heap.  I do have worries about the efficiency of this though:  
> when marshaling a closure across a socket to another process, it  
> must marshal all the necessary state as well.  I would imagine this  
> has the potential to get slow if there was much state to marshal  
> (i.e., a closure marshaled referencing a large data structure).
>
> --Jonathan
>
>>
>> Erik
>> -- 
>> -----------------------------------------------------------------
>> Erik de Castro Lopo
>> -----------------------------------------------------------------
>> "... a discussion of C++'s strengths and flaws always sounds
>> like an argument about whether one should face north or east
>> when one is sacrificing one's goat to the rain god."
>> -- Thant Tessman
>>
>> _______________________________________________
>> Caml-list mailing list. Subscription management:
>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>> Archives: http://caml.inria.fr
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: Fwd: [Caml-list] Faking concurrency using Unix forks and pipes

[Erik de Castro Lopo]

Jonathan Bryant wrote:

> What I've always done to get around this is fork a process at the  
> very beginning of the code, and whenever I need to fork a process  
> later, I fork one from that process.  That way almost nothing is in  
> the heap.

Yep, that works. Its also probably something that could be built 
into the JoCaml extension.

>  I do have worries about the efficiency of this though:  
> when marshaling a closure across a socket to another process, it  
> must marshal all the necessary state as well.  I would imagine this  
> has the potential to get slow if there was much state to marshal  
> (i.e., a closure marshaled referencing a large data structure).

This is a potential problem. This is like so many language features
that if misused, can lead to huge performance degradations. Knowing
about the problem beforehand allows programmers to avoid it.

Erik
-- 
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"If you have an apple and I have  an apple and we  exchange apples then
you and I will still each have  one apple. But  if you have an idea and I
have an idea and we exchange these ideas, then each of us will have two
ideas." -- George Bernard Shaw

Re: [Caml-list] Faking concurrency using Unix forks and pipes

[Jonathan Bryant]

>
>>  I do have worries about the efficiency of this though:
>> when marshaling a closure across a socket to another process, it
>> must marshal all the necessary state as well.  I would imagine this
>> has the potential to get slow if there was much state to marshal
>> (i.e., a closure marshaled referencing a large data structure).
>
> This is a potential problem. This is like so many language features
> that if misused, can lead to huge performance degradations. Knowing
> about the problem beforehand allows programmers to avoid it.

Just throwing out an idea, so someone who is more familiar with the  
GC/runtime correct me if this can't be done, but what about a smaller  
separate heap in shared memory that is reserved for concurrent data?   
Maybe you could declare a type

stype 'a tree =
| Node of 'a * 'a tree * 'a tree
| Leaf of 'a

that is always allocated in the shared heap, and have a separate GC  
thread that manages that heap.  Could that GC be concurrent without  
affecting the performance of non-concurrent data?

--Jonathan

Re: [Caml-list] Faking concurrency using Unix forks and pipes

[Jon Harrop]

On Wednesday 30 May 2007 21:05:11 Jonathan Bryant wrote:
> Just throwing out an idea, so someone who is more familiar with the
> GC/runtime correct me if this can't be done, but what about a smaller
> separate heap in shared memory that is reserved for concurrent data?
> Maybe you could declare a type
>
> stype 'a tree =
>
> | Node of 'a * 'a tree * 'a tree
> | Leaf of 'a
>
> that is always allocated in the shared heap, and have a separate GC
> thread that manages that heap.  Could that GC be concurrent without
> affecting the performance of non-concurrent data?

The programmer would need to explicitly move data from one heap to another, 
which undermines the high-level nature of a functional programming language. 
You'd be back to C-style memory allocation.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e