Co-existing with non OCaml threads

Co-existing with non OCaml threads

[Francois Rouaix]

[-- Attachment #1: Type: text/plain, Size: 820 bytes --]

I'm contemplating writing an OCaml interface for a C++ middleware library
that my company develops and uses internally. Typically this middleware will
start an event loop on a thread in the background, leaving the application
responsible for its own threads (and potentially using none and having its
own code running entirely on events within the eventloop thread).
How's this likely to be compatible with OCaml use of native threads (this is
on Linux by the way)?
The manual section for interfacing with C isn't mentionning threads
anywhere.
Should Caml code be restricted to run on threads it has created? Or can it
run on any threads?
How can I synchronize between a thread running C++ code and a thread running
OCaml code (i.e. both communicating on a message queue)?
Thanks for any suggestions.
--f

[-- Attachment #2: Type: text/html, Size: 836 bytes --]

Re: [Caml-list] Co-existing with non OCaml threads

[Ted Kremenek]

Hi Francois,

 From my understanding of OCaml threads, they are non-preemptive user- 
level threads that run within a single kernel thread.  Further, OCaml  
does not employ a concurrent garbage collector; essentially the whole  
program stops (including all OCaml threads) when the garbage  
collector runs.  OCaml threads do not provide any additional actual  
parallelism (they are not preemptive like kernel threads), but like  
all user-level threads they provide an abstraction of parallel  
threads as a programming idiom.  I am not certain if in the  
implementation of OCaml threads that one thread automatically yields  
to another when it calls a blocking operation like an I/O operation  
(like the user-level thread implementation for C described in the  
Capriccio paper that appeared in SOSP a few years ago), but I doubt it.

You might be able to run the OCaml run-time (I am talking about  
native code) within a program that employs multiple kernel threads as  
long as ALL the OCaml code runs in one designated thread, but the run- 
time was not designed to be executed within multiple kernel threads.

Of course I could be completely wrong about this, but this was my  
interpretation from the documentation, previous emails on this list,  
and from my brief circumspection of the code from the OCaml run-time.

Ted

On May 11, 2006, at 5:29 PM, Francois Rouaix wrote:

> I'm contemplating writing an OCaml interface for a C++ middleware  
> library that my company develops and uses internally. Typically  
> this middleware will start an event loop on a thread in the  
> background, leaving the application responsible for its own threads  
> (and potentially using none and having its own code running  
> entirely on events within the eventloop thread).
> How's this likely to be compatible with OCaml use of native threads  
> (this is on Linux by the way)?
> The manual section for interfacing with C isn't mentionning threads  
> anywhere.
> Should Caml code be restricted to run on threads it has created? Or  
> can it run on any threads?
> How can I synchronize between a thread running C++ code and a  
> thread running OCaml code (i.e. both communicating on a message  
> queue)?
> Thanks for any suggestions.
> --f
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Co-existing with non OCaml threads

[Gerd Stolpmann]

Am Donnerstag, den 11.05.2006, 17:29 -0700 schrieb Francois Rouaix:
> I'm contemplating writing an OCaml interface for a C++ middleware
> library that my company develops and uses internally. Typically this
> middleware will start an event loop on a thread in the background,
> leaving the application responsible for its own threads (and
> potentially using none and having its own code running entirely on
> events within the eventloop thread). 
> How's this likely to be compatible with OCaml use of native threads
> (this is on Linux by the way)?
> The manual section for interfacing with C isn't mentionning threads
> anywhere.
> Should Caml code be restricted to run on threads it has created? Or
> can it run on any threads? 
> How can I synchronize between a thread running C++ code and a thread
> running OCaml code (i.e. both communicating on a message queue)?
> Thanks for any suggestions.

If you compile ocaml with pthread support, the O'Caml threads are real
Linux threads. When using them, be aware of:

- The memory management code is not thread-safe. That means 
  only one thread is allowed to run O'Caml code at any time.
  To ensure this there is the so-called "master lock". You
  can acquire the master lock by calling

  leave_blocking_section()

  and you can release it by

  enter_blocking_section()

  When you call a C function that may block for indefinite time
  one usually releases the lock first, calls the function, and
  then requires it. The stub function looks like

  value foo_stub (...) {
     ...
     enter_blocking_section();
     /* From here: NO ACCESS to any O'Caml memory, not even
      * CAMLparam/CAMLlocal variables!
      */
     ...
     foo();
     ...
     leave_blocking_section();
     /* Normal rules for stub functions apply again */
     ...
  }

  For callbacks from C code you will typically first call 
  leave_blocking_section() and later enter_blocking_section()
  before you return.

- O'Caml must never see threads not created by itself. This 
  means a thread created by your middleware must not run
  O'Caml code. (You get immediately a segfault if you try.)

- Although O'Caml mutexes and condition variables actually
  base on their pthread counterparts, there is no interface
  from the C side. So you better do any synchronization 
  completely in C, i.e. the message queue is written in C
  and uses the normal pthread primitives. The O'Caml stub
  functions accessing this queue need to call
  enter/leave_blocking_section as explained.

Hope this helps. Had to solve a similar problem for a customer.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: Co-existing with non OCaml threads

[Vincenzo Ciancia]

Gerd Stolpmann wrote:

> - O'Caml must never see threads not created by itself. This
> means a thread created by your middleware must not run
> O'Caml code. (You get immediately a segfault if you try.)
> 

However, let's make it clearer, you can create a thread in ocaml and then
call a C function, and this C function, which will be run concurrently with
other C functions called the same way, can call back ocaml.

Vincenzo

-- 
Please note that I do not read the e-mail address used in the from field but
I read vincenzo_ml at yahoo dot it
Attenzione: non leggo l'indirizzo di posta usato nel campo from, ma leggo
vincenzo_ml at yahoo dot it

Re: [Caml-list] Re: Co-existing with non OCaml threads

[Markus Mottl]

On 5/12/06, Vincenzo Ciancia <vincenzo_yahoo_addressguard-gmane@yahoo.it> wrote:
> However, let's make it clearer, you can create a thread in ocaml and then
> call a C function, and this C function, which will be run concurrently with
> other C functions called the same way, can call back ocaml.

Yes, this is possible, and good libraries provide ways of letting user
created threads enter the library to execute callbacks.

Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Co-existing with non OCaml threads

[Ted Kremenek]

This is very interesting.  Thank you for pointing this out.  I have  
some questions that would clarify a few things for me.

Because of the run-time lock, should I gather that the threads are  
still essentially cooperative threads?  For example, does it work  
this way: if a thread holding the master lock is switched out and the  
kernel schedules another thread, that other thread will start running  
and try and acquire the lock.  It won't be able to, so it goes back  
to sleep, and another thread will wake up, try and acquire the lock,  
goes back to sleep, and so on, until the original thread holding the  
lock is rescheduled.  Only when the thread releases the lock  
(yields?) will another thread be able to run.  Is this how it works?   
If so, this would lend itself to extremely poor performance: if I had  
100 threads, 99 of them may have to wake up and go to sleep before  
the original one is scheduled.  That is 99 useless context switches.   
Or rather is the lock acquired and released (within the generated  
native code for the OCaml part of a program, not C code) on calls to  
the memory management functions and other run-time code that are not  
thread-safe?  This is also seems slow, since the heap is actively  
manipulated all the time, so locks will constantly be acquired and  
released, and you will have the same wake-up, make little or no  
progress and go back to sleep problem I mentioned before.

Your last email said that only one thread is allowed to run OCaml  
code at any time, so it seems to me that this mutual exclusion must  
come from somewhere.  I'm very curious to know how this is  
implemented.  I gather that most people want to use threads in OCaml  
to have multiple threads running OCaml code, and not necessarily have  
a bunch of threads executing called C code (allowing the master lock  
to be released).  I'm just trying to understand how actual  
performance would ever resemble anything desirable.

Just curious.  Thanks for the informative email.

Ted

On May 12, 2006, at 4:22 AM, Gerd Stolpmann wrote:

> If you compile ocaml with pthread support, the O'Caml threads are real
> Linux threads. When using them, be aware of:
>
> - The memory management code is not thread-safe. That means
>   only one thread is allowed to run O'Caml code at any time.
>   To ensure this there is the so-called "master lock". You
>   can acquire the master lock by calling
>
>   leave_blocking_section()
>
>   and you can release it by
>
>   enter_blocking_section()

Re: [Caml-list] Co-existing with non OCaml threads

[Gerd Stolpmann]

Am Freitag, den 12.05.2006, 10:43 -0700 schrieb Ted Kremenek:
> This is very interesting.  Thank you for pointing this out.  I have  
> some questions that would clarify a few things for me.
> 
> Because of the run-time lock, should I gather that the threads are  
> still essentially cooperative threads?  

Essentially they are pthreads, i.e. on Linux kernel threads. However,
there is a level of cooperation on top of this.

> For example, does it work  
> this way: if a thread holding the master lock is switched out and the  
> kernel schedules another thread, that other thread will start running  
> and try and acquire the lock.

For all threads known to O'Caml we have:

- Either the thread is running
- Or it is waiting for the master lock (the kernel knows that, 
  and won't schedule this thread)
- Or it is executing non-O'Caml code because it passed
  enter_blocking_section.

I hope it is now clear that getting the master lock is not the problem.
The trick is how a thread releases it.

The running thread (that has the lock) is notified when it must release
the master lock. There is a special controller thread that periodically
checks whether the running thread ran too long. If so, a special
notification mechanism will cause that this thread gives the master lock
up.

Effectively, this is a second scheduler on top of the kernel scheduler.

>   It won't be able to, so it goes back  
> to sleep, and another thread will wake up, try and acquire the lock,  
> goes back to sleep, and so on, until the original thread holding the  
> lock is rescheduled.  

No, this won't happen, because the kernel does not wake up threads
waiting for a lock that is not free. These are kernel-level locks!

> Only when the thread releases the lock  
> (yields?) will another thread be able to run.  Is this how it works?   
> If so, this would lend itself to extremely poor performance: if I had  
> 100 threads, 99 of them may have to wake up and go to sleep before  ie..
> the original one is scheduled.  That is 99 useless context switches.   
> Or rather is the lock acquired and released (within the generated  
> native code for the OCaml part of a program, not C code) on calls to  
> the memory management functions and other run-time code that are not  
> thread-safe?

No this is not done.

>   This is also seems slow, since the heap is actively  
> manipulated all the time, so locks will constantly be acquired and  
> released, and you will have the same wake-up, make little or no  
> progress and go back to sleep problem I mentioned before.
> 
> Your last email said that only one thread is allowed to run OCaml  
> code at any time, so it seems to me that this mutual exclusion must  
> come from somewhere.  I'm very curious to know how this is  
> implemented.  I gather that most people want to use threads in OCaml  
> to have multiple threads running OCaml code, and not necessarily have  
> a bunch of threads executing called C code (allowing the master lock  
> to be released).  I'm just trying to understand how actual  
> performance would ever resemble anything desirable.

Performance is quite good, but you should keep in mind:

- O'Caml code can never run on more than one CPU

- Switches between O'Caml threads are less fine-grained than 
  between kernel threads. Imagine you stat a file, and it is
  necessary to load some blocks from disk. This can take
  0.01 s of time. During that time a switch is impossible.
  (I.e. the problem is that there are system calls that are
  non-blocking in general but that can nevertheless consume
  lots of time in unlucky data cases.)

Gerd

> 
> Just curious.  Thanks for the informative email.
> 
> Ted
> 
> On May 12, 2006, at 4:22 AM, Gerd Stolpmann wrote:
> 
> > If you compile ocaml with pthread support, the O'Caml threads are real
> > Linux threads. When using them, be aware of:
> >
> > - The memory management code is not thread-safe. That means
> >   only one thread is allowed to run O'Caml code at any time.
> >   To ensure this there is the so-called "master lock". You
> >   can acquire the master lock by calling
> >
> >   leave_blocking_section()
> >
> >   and you can release it by
> >
> >   enter_blocking_section()
> 
> 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Co-existing with non OCaml threads

[Ted Kremenek]

Gerd,

Excellent.  I assumed that there was  second scheduler of some sort,  
but it wasn't clear how it was done.  Thank you very much for  
elucidating this.

Ted

On May 12, 2006, at 11:51 AM, Gerd Stolpmann wrote:

> Am Freitag, den 12.05.2006, 10:43 -0700 schrieb Ted Kremenek:
>> This is very interesting.  Thank you for pointing this out.  I have
>> some questions that would clarify a few things for me.
>>
>> Because of the run-time lock, should I gather that the threads are
>> still essentially cooperative threads?
>
> Essentially they are pthreads, i.e. on Linux kernel threads. However,
> there is a level of cooperation on top of this.
>
>> For example, does it work
>> this way: if a thread holding the master lock is switched out and the
>> kernel schedules another thread, that other thread will start running
>> and try and acquire the lock.
>
> For all threads known to O'Caml we have:
>
> - Either the thread is running
> - Or it is waiting for the master lock (the kernel knows that,
>   and won't schedule this thread)
> - Or it is executing non-O'Caml code because it passed
>   enter_blocking_section.
>
> I hope it is now clear that getting the master lock is not the  
> problem.
> The trick is how a thread releases it.
>
> The running thread (that has the lock) is notified when it must  
> release
> the master lock. There is a special controller thread that  
> periodically
> checks whether the running thread ran too long. If so, a special
> notification mechanism will cause that this thread gives the master  
> lock
> up.
>
> Effectively, this is a second scheduler on top of the kernel  
> scheduler.
>
>>   It won't be able to, so it goes back
>> to sleep, and another thread will wake up, try and acquire the lock,
>> goes back to sleep, and so on, until the original thread holding the
>> lock is rescheduled.
>
> No, this won't happen, because the kernel does not wake up threads
> waiting for a lock that is not free. These are kernel-level locks!
>
>> Only when the thread releases the lock
>> (yields?) will another thread be able to run.  Is this how it works?
>> If so, this would lend itself to extremely poor performance: if I had
>> 100 threads, 99 of them may have to wake up and go to sleep  
>> before  ie..
>> the original one is scheduled.  That is 99 useless context switches.
>> Or rather is the lock acquired and released (within the generated
>> native code for the OCaml part of a program, not C code) on calls to
>> the memory management functions and other run-time code that are not
>> thread-safe?
>
> No this is not done.
>
>>   This is also seems slow, since the heap is actively
>> manipulated all the time, so locks will constantly be acquired and
>> released, and you will have the same wake-up, make little or no
>> progress and go back to sleep problem I mentioned before.
>>
>> Your last email said that only one thread is allowed to run OCaml
>> code at any time, so it seems to me that this mutual exclusion must
>> come from somewhere.  I'm very curious to know how this is
>> implemented.  I gather that most people want to use threads in OCaml
>> to have multiple threads running OCaml code, and not necessarily have
>> a bunch of threads executing called C code (allowing the master lock
>> to be released).  I'm just trying to understand how actual
>> performance would ever resemble anything desirable.
>
> Performance is quite good, but you should keep in mind:
>
> - O'Caml code can never run on more than one CPU
>
> - Switches between O'Caml threads are less fine-grained than
>   between kernel threads. Imagine you stat a file, and it is
>   necessary to load some blocks from disk. This can take
>   0.01 s of time. During that time a switch is impossible.
>   (I.e. the problem is that there are system calls that are
>   non-blocking in general but that can nevertheless consume
>   lots of time in unlucky data cases.)
>
> Gerd
>
>>
>> Just curious.  Thanks for the informative email.
>>
>> Ted
>>
>> On May 12, 2006, at 4:22 AM, Gerd Stolpmann wrote:
>>
>>> If you compile ocaml with pthread support, the O'Caml threads are  
>>> real
>>> Linux threads. When using them, be aware of:
>>>
>>> - The memory management code is not thread-safe. That means
>>>   only one thread is allowed to run O'Caml code at any time.
>>>   To ensure this there is the so-called "master lock". You
>>>   can acquire the master lock by calling
>>>
>>>   leave_blocking_section()
>>>
>>>   and you can release it by
>>>
>>>   enter_blocking_section()
>>
>>
> -- 
> ------------------------------------------------------------
> Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany
> gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
> Phone: +49-6151-153855                  Fax: +49-6151-997714
> ------------------------------------------------------------
>

Re: [Caml-list] Co-existing with non OCaml threads

[Markus Mottl]

On 5/12/06, Ted Kremenek <kremenek@cs.stanford.edu> wrote:
> Because of the run-time lock, should I gather that the threads are
> still essentially cooperative threads?  For example, does it work
> this way: if a thread holding the master lock is switched out and the
> kernel schedules another thread, that other thread will start running
> and try and acquire the lock.  It won't be able to, so it goes back
> to sleep, and another thread will wake up, try and acquire the lock,
> goes back to sleep, and so on, until the original thread holding the
> lock is rescheduled.

No.  The other threads will block in the master lock, which is a
POSIX-mutex + condition variable.  Releasing the master lock will
signal exactly one of the blocking threads that it may run.  The
context switch, however, does not necessarily happen immediately.  It
depends on the OS when this will happen, and the scheduling policy
determines which thread is going to run.

> If so, this would lend itself to extremely poor performance: if I had
> 100 threads, 99 of them may have to wake up and go to sleep before
> the original one is scheduled.  That is 99 useless context switches.

That would be horrible, and that's not the way it works.

> Or rather is the lock acquired and released (within the generated
> native code for the OCaml part of a program, not C code) on calls to
> the memory management functions and other run-time code that are not
> thread-safe?  This is also seems slow, since the heap is actively
> manipulated all the time, so locks will constantly be acquired and
> released, and you will have the same wake-up, make little or no
> progress and go back to sleep problem I mentioned before.

A timer signal makes sure that the current thread yields once in a
while.  Thus, it is not necessary to acquire/release locks at each
allocation, which would make everything run dog slow.

> Your last email said that only one thread is allowed to run OCaml
> code at any time, so it seems to me that this mutual exclusion must
> come from somewhere.  I'm very curious to know how this is
> implemented.  I gather that most people want to use threads in OCaml
> to have multiple threads running OCaml code, and not necessarily have
> a bunch of threads executing called C code (allowing the master lock
> to be released).  I'm just trying to understand how actual
> performance would ever resemble anything desirable.

Unless INRIA implements a GC that can handle multiple threads running
in parallel, which would have its own performance tradeoffs, you'll
essentially always share one processor only.  It depends on your
application whether that's a problem.  I/O-heavy applications will do
fine, because the system calls can all be performed in parallel.  You
can also always call C or Fortran to run in parallel on bigarrays
(matrices), because they don't mess with the OCaml-heap.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

Re: [Caml-list] Co-existing with non OCaml threads

[Ted Kremenek]

Thanks Markus.  I think my main confusion was I forgot the lock was  
managed by the kernel (since it is managed through the pthreads API)  
and that the notify would be on a single blocked thread.  If the  
signaling was a broadcast where all blocked threads were moved to the  
ready queue (versus waking up a single thread) the problem I  
mentioned would still appear.  I.e., if all the blocked threads woke  
up, only one would acquire the lock, but the remaining would get  
scheduled and block.  Fortunately a broadcast isn't needed for this  
problem, since any waiting thread can be rescheduled.  I should have  
realized this on my own, but I appreciate the description of  
"secondary" scheduler implemented by the timer mechanism.

On May 12, 2006, at 1:44 PM, Markus Mottl wrote:

> On 5/12/06, Ted Kremenek <kremenek@cs.stanford.edu> wrote:
>> Because of the run-time lock, should I gather that the threads are
>> still essentially cooperative threads?  For example, does it work
>> this way: if a thread holding the master lock is switched out and the
>> kernel schedules another thread, that other thread will start running
>> and try and acquire the lock.  It won't be able to, so it goes back
>> to sleep, and another thread will wake up, try and acquire the lock,
>> goes back to sleep, and so on, until the original thread holding the
>> lock is rescheduled.
>
> No.  The other threads will block in the master lock, which is a
> POSIX-mutex + condition variable.  Releasing the master lock will
> signal exactly one of the blocking threads that it may run.  The
> context switch, however, does not necessarily happen immediately.  It
> depends on the OS when this will happen, and the scheduling policy
> determines which thread is going to run.
>
>> If so, this would lend itself to extremely poor performance: if I had
>> 100 threads, 99 of them may have to wake up and go to sleep before
>> the original one is scheduled.  That is 99 useless context switches.
>
> That would be horrible, and that's not the way it works.
>
>> Or rather is the lock acquired and released (within the generated
>> native code for the OCaml part of a program, not C code) on calls to
>> the memory management functions and other run-time code that are not
>> thread-safe?  This is also seems slow, since the heap is actively
>> manipulated all the time, so locks will constantly be acquired and
>> released, and you will have the same wake-up, make little or no
>> progress and go back to sleep problem I mentioned before.
>
> A timer signal makes sure that the current thread yields once in a
> while.  Thus, it is not necessary to acquire/release locks at each
> allocation, which would make everything run dog slow.
>
>> Your last email said that only one thread is allowed to run OCaml
>> code at any time, so it seems to me that this mutual exclusion must
>> come from somewhere.  I'm very curious to know how this is
>> implemented.  I gather that most people want to use threads in OCaml
>> to have multiple threads running OCaml code, and not necessarily have
>> a bunch of threads executing called C code (allowing the master lock
>> to be released).  I'm just trying to understand how actual
>> performance would ever resemble anything desirable.
>
> Unless INRIA implements a GC that can handle multiple threads running
> in parallel, which would have its own performance tradeoffs, you'll
> essentially always share one processor only.  It depends on your
> application whether that's a problem.  I/O-heavy applications will do
> fine, because the system calls can all be performed in parallel.  You
> can also always call C or Fortran to run in parallel on bigarrays
> (matrices), because they don't mess with the OCaml-heap.
>
> Regards,
> Markus
>
> -- 
> Markus Mottl        http://www.ocaml.info         
> markus.mottl@gmail.com

Re: [Caml-list] Co-existing with non OCaml threads

[Xavier Leroy]

Hi François,

> [Interfacing Caml code with multithreaded C/C++ code]
> The manual section for interfacing with C isn't mentionning threads
> anywhere.  Should Caml code be restricted to run on threads it has
> created? Or can it run on any threads?

It depends whether your Caml code is single-threaded or
multi-threaded.

Case 1: The Caml code is single-threaded (more precisely: it is not
linked with Caml's threading library).  For instance, Caml provides a
number of functions that you will call back from the main C/C++
multithreaded program.  In this case, you can call back Caml code from
any thread.  All you need to make sure is that you never call back from
two threads concurrently.  Just put a lock around the callbacks and
you're done.

Case 2: The Caml code is linked with Caml's threading library.
Then, you can only call back from threads created by Caml, and as Gerd
Stolpmann mentioned, there are additional subtleties to ensure mutual
exclusion.  I'd recommend you avoid this scenario.

> How can I synchronize between a thread running C++ code and a thread
> running OCaml code (i.e. both communicating on a message queue)?

If your message queue is already implemented in C++, all you need to
do is wrap the "put" and "get" operations of the message queue using
the Caml foreign-function interface, and call these functions from
your Caml code.

Hope this helps,

- Xavier