[Caml-list] Native threads under Debian 3.0r0

[Caml-list] Native threads under Debian 3.0r0

[Berke Durak]

Hello everybody,

I'm having severe problems with native threads. Not only I can't get them to
work with the Graphics module, I can't get them to work at all. I'm
using a fresh Debian 3.0r0. I've recompiled Ocaml 3.06 with
'-with-pthread' option.  Shouldn't the following code

$ cat toto.ml
open Thread

let _ =
  let code1 () =
	let x = ref 0 in
	while true do
	  incr x;
	  Printf.printf "alpha %d\n" !x;
	  flush stdout
	done
  in
  let thread1 = Thread.create code1 ()
  in
  while true do
	()
  done

compiled with

$ ocamlopt.opt -thread unix.cmxa threads.cmxa toto.ml -o toto

and run as

$ ./toto

give me something like

alpha 0
alpha 1
alpha 2
alpha 3
alpha 4
alpha 5
alpha 6
...

?

Instead, the program doesn't output anything, and when I hit Ctrl-C,
it leaves some processes running, which can only be killed with
SIGKILL.

The program works when compiled as bytecode. System info :

$ uname -a
Linux gogol 2.4.18 #1 Thu Aug 22 20:42:28 CEST 2002 i586 unknown
$ ls -l /lib/libpthread*
-rw-r--r--    1 root     root       102172 Apr 28 11:57 /lib/libpthread-0.9.so

However, even in bytecode, when I use the Graphics module, some subprocesses
refuse to die when I interrupt the main program.

Any clues ? Thanks very much.
--
Berke Durak
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Markus Mottl]

On Wed, 04 Sep 2002, Berke Durak wrote:
> I'm having severe problems with native threads. Not only I can't get them to
> work with the Graphics module, I can't get them to work at all. I'm
> using a fresh Debian 3.0r0. I've recompiled Ocaml 3.06 with
> '-with-pthread' option.  Shouldn't the following code

(cross-posted to caml-bugs)

There indeed seems to be something broken about threads (at least native
ones). Things work fine with byte-code and the toplevel, but native
code + native threads don't.  I have managed to reproduced this problem
under Linux and Solaris with OCaml 3.06, but also with an older version
(3.04+10 - 2002-04-18) on an Alpha (Digital Unix).

Regards,
Markus Mottl

> $ cat toto.ml
> open Thread
> 
> let _ =
>   let code1 () =
> 	let x = ref 0 in
> 	while true do
> 	  incr x;
> 	  Printf.printf "alpha %d\n" !x;
> 	  flush stdout
> 	done
>   in
>   let thread1 = Thread.create code1 ()
>   in
>   while true do
> 	()
>   done
> 
> compiled with
> 
> $ ocamlopt.opt -thread unix.cmxa threads.cmxa toto.ml -o toto
> 
> and run as
> 
> $ ./toto
> 
> give me something like
> 
> alpha 0
> alpha 1
> alpha 2
> alpha 3
> alpha 4
> alpha 5
> alpha 6
> ...
> 
> ?
> 
> Instead, the program doesn't output anything, and when I hit Ctrl-C,
> it leaves some processes running, which can only be killed with
> SIGKILL.
> 
> The program works when compiled as bytecode. System info :
> 
> $ uname -a
> Linux gogol 2.4.18 #1 Thu Aug 22 20:42:28 CEST 2002 i586 unknown
> $ ls -l /lib/libpthread*
> -rw-r--r--    1 root     root       102172 Apr 28 11:57 /lib/libpthread-0.9.so
> 
> However, even in bytecode, when I use the Graphics module, some subprocesses
> refuse to die when I interrupt the main program.
> 
> Any clues ? Thanks very much.
> --
> Berke Durak
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Jacques Garrigue]

From: Markus Mottl <markus@oefai.at>
> 
> There indeed seems to be something broken about threads (at least native
> ones). Things work fine with byte-code and the toplevel, but native
> code + native threads don't.  I have managed to reproduced this problem
> under Linux and Solaris with OCaml 3.06, but also with an older version
> (3.04+10 - 2002-04-18) on an Alpha (Digital Unix).

This behaviour is normal, and I believe documented.
Looking around line 270 in otherlibs/systhreads/posix.c gives the
explanation.
You cannot switch between caml threads just anywhere, you
have to choose a safe place to do that. The way to do that is to raise
a flag requesting a thread change, and wait for the code to actually
release control.
In bytecode "something_to_do" is used, and checked pretty often: at
every function call and every loop iteration.
However, this would mean too much overhead for native code, so
switching is only checked when the GC is called. To make it work even
on slowly allocating programs, the young generation is declared "full"
every once in a while, but there must still be some allocation for it to
happen.

So the answer is: a native code thread should be allocating some
data, otherwise it cannot be interrupted. The problem does not exist
for bytecode.

---------------------------------------------------------------------------
Jacques Garrigue      Kyoto University     garrigue at kurims.kyoto-u.ac.jp
		<A HREF=http://wwwfun.kurims.kyoto-u.ac.jp/~garrigue/>JG</A>


> > $ cat toto.ml
> > open Thread
> > 
> > let _ =
> >   let code1 () =
> > 	let x = ref 0 in
> > 	while true do
> > 	  incr x;
> > 	  Printf.printf "alpha %d\n" !x;
> > 	  flush stdout
> > 	done
> >   in
> >   let thread1 = Thread.create code1 ()
> >   in
> >   while true do
> > 	()
> >   done
> > 
> > compiled with
> > 
> > $ ocamlopt.opt -thread unix.cmxa threads.cmxa toto.ml -o toto
> > 
> > and run as
> > 
> > $ ./toto
> > 
> > give me something like
> > 
> > alpha 0
> > alpha 1
> > alpha 2
> > alpha 3
> > alpha 4
> > alpha 5
> > alpha 6
> > ...
> > 
> > ?
> > 
> > Instead, the program doesn't output anything, and when I hit Ctrl-C,
> > it leaves some processes running, which can only be killed with
> > SIGKILL.
> > 
> > The program works when compiled as bytecode. System info :
> > 
> > $ uname -a
> > Linux gogol 2.4.18 #1 Thu Aug 22 20:42:28 CEST 2002 i586 unknown
> > $ ls -l /lib/libpthread*
> > -rw-r--r--    1 root     root       102172 Apr 28 11:57 /lib/libpthread-0.9.so
> > 
> > However, even in bytecode, when I use the Graphics module, some subprocesses
> > refuse to die when I interrupt the main program.
> > 
> > Any clues ? Thanks very much.
> > --
> > Berke Durak
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Olivier Andrieu]

 Jacques Garrigue [Wednesday 4 September 2002] :
 > So the answer is: a native code thread should be allocating some
 > data, otherwise it cannot be interrupted. 

Or he should call Thread.yield (), right ?


-- 
   Olivier
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Markus Mottl]

On Wed, 04 Sep 2002, Jacques Garrigue wrote:
> In bytecode "something_to_do" is used, and checked pretty often:
> at every function call and every loop iteration.

Ok.

> However, this would mean too much overhead for native code, so switching
> is only checked when the GC is called.

Actually, calling the GC manually does not make any difference. The only
thing that helps is calling "Thread.yield" in the infinite loop.

> To make it work even on slowly allocating programs, the young generation
> is declared "full" every once in a while, but there must still be some
> allocation for it to happen.

I see. So if the GC doesn't really have to do any work, it will also
not switch threads even when called manually.

> So the answer is: a native code thread should be allocating some data,
> otherwise it cannot be interrupted. The problem does not exist for
> bytecode.

The main consideration is obviously native code performance. OTOH, some
people might want to guarantee that their threads never get stuck. I
don't know what solution would be useful for them. Maybe some runtime
or command-line switch that trades efficiency against such scheduling
problems.

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Berke Durak]

On Wed, Sep 04, 2002 at 03:01:51PM +0200, Markus Mottl wrote:
> On Wed, 04 Sep 2002, Jacques Garrigue wrote:
>
> > So the answer is: a native code thread should be allocating some data,
> > otherwise it cannot be interrupted. The problem does not exist for
> > bytecode.
> 
> The main consideration is obviously native code performance. OTOH, some
> people might want to guarantee that their threads never get stuck. I
> don't know what solution would be useful for them. Maybe some runtime
> or command-line switch that trades efficiency against such scheduling
> problems.

Sorry about my ignorance, but aren't threads scheduled by the kernel
when using libpthread ? (That would make sense, since I get extra
processes when I create extra threads). I mean, aren't the threads
automatically preempted ?

If not, couldn't it be possible to preempt them automatically using
the interval timer ?
--
Berke
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Jacques Garrigue]

From: Berke Durak <berke@altern.org>
> On Wed, Sep 04, 2002 at 03:01:51PM +0200, Markus Mottl wrote:
> > On Wed, 04 Sep 2002, Jacques Garrigue wrote:
> >
> > > So the answer is: a native code thread should be allocating some data,
> > > otherwise it cannot be interrupted. The problem does not exist for
> > > bytecode.
> > 
> > The main consideration is obviously native code performance. OTOH, some
> > people might want to guarantee that their threads never get stuck. I
> > don't know what solution would be useful for them. Maybe some runtime
> > or command-line switch that trades efficiency against such scheduling
> > problems.

Not impossible to do, but this is a design question.
I expect you would just have to insert explicit checks for pending
signals like in bytecode: at function calls and in loops.

Of course it is always possible to call Thread.yield by hand, but I'm
a bit afraid of the extra overhead since this is a system call.
And it may mean your process is executed for shorter periods.

Note that there is only a real problem for long running loops doing
purely numerical computation without any allocation. And inserting an
allocating function call somewhere in an outer loop should solve the
problem: ignore ([1] @ [2]) for instance.

> Sorry about my ignorance, but aren't threads scheduled by the kernel
> when using libpthread ? (That would make sense, since I get extra
> processes when I create extra threads). I mean, aren't the threads
> automatically preempted ?
> 
> If not, couldn't it be possible to preempt them automatically using
> the interval timer ?

This is exactly what is done. However such a preempted thread will not
release its caml mutex, which makes sure allocation doesn't occur in
two threads simultaneoulsy for instance. So C threads will run, but
not other caml threads, since they would have to get the mutex.
As I tried to explain in my previous mail, the interval timer lets the
current thread know that it should free the mutex as soon as possible,
but this is the responsibility of the running thread to do so.

Jacques Garrigue
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Sven LUTHER]

On Wed, Sep 04, 2002 at 01:03:14AM +0200, Berke Durak wrote:
> Hello everybody,
> 
> I'm having severe problems with native threads. Not only I can't get them to
> work with the Graphics module, I can't get them to work at all. I'm
> using a fresh Debian 3.0r0. I've recompiled Ocaml 3.06 with
> '-with-pthread' option.  Shouldn't the following code

Notice that you should be able to install the sid version of the
packages with something like :

apt-get install -t sid ocaml

(and -t sarge when ocaml 3.06 packages enter testing in a few days).

So you don't need to recompile yourself to get ocaml 3.06 and the other
library packages you are using.

That said, the problem you mention below (if it is a problem) is also present
with the 3.06 debian packages.

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Stefano Zacchiroli]

On Wed, Sep 04, 2002 at 04:38:44PM +0200, Sven LUTHER wrote:
> Notice that you should be able to install the sid version of the
> packages with something like :
> 
> apt-get install -t sid ocaml

Not really, ocaml package on sid depends on a libc6 version that isn't
available on woody (Debian 3.0r0).

I have set up an apt-gettable repository with various ocaml packages
rebuild for woody, if you need them, just ask.

Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Sven LUTHER]

On Thu, Sep 05, 2002 at 11:17:09AM +0200, Stefano Zacchiroli wrote:
> On Wed, Sep 04, 2002 at 04:38:44PM +0200, Sven LUTHER wrote:
> > Notice that you should be able to install the sid version of the
> > packages with something like :
> > 
> > apt-get install -t sid ocaml
> 
> Not really, ocaml package on sid depends on a libc6 version that isn't
> available on woody (Debian 3.0r0).

Yes, i forgot that.

BTW, would the -t sid option not also install the dependencies, and
thus upgrade libc6, which should work painlessly ?

> I have set up an apt-gettable repository with various ocaml packages
> rebuild for woody, if you need them, just ask.

Mmm, should we provide an official version of those ?
Do you know if it is possible to easily upload packages for
woody-proposed-updates or something such ?

Friendly,

Sven Luther
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Native threads under Debian 3.0r0

[Stefano Zacchiroli]

[-- Attachment #1: Type: text/plain, Size: 1464 bytes --]

On Thu, Sep 05, 2002 at 12:42:59PM +0200, Sven LUTHER wrote:
> BTW, would the -t sid option not also install the dependencies, and
> thus upgrade libc6, which should work painlessly ?

In fact I don't know exactly how the new -t and apt-pinning stuff work,
but even if it download all the dependencies probably is not a good idea
to upgrade libc6 to a sid version for a stable/debian. Is better to
rebuild the packages.

> > I have set up an apt-gettable repository with various ocaml packages
> > rebuild for woody, if you need them, just ask.
> 
> Mmm, should we provide an official version of those ?

I'm working on it, I've just finished (up to debugging) an almost
automatic mechanism to rebuild ocaml related packages from sid to woody.
When all will be up and running I'm planning to advertise an official
(limited to the debian ocaml community) apt-gettable repository of
rebuilt packages.

> Do you know if it is possible to easily upload packages for
> woody-proposed-updates or something such ?

Don't know, I have planned to read the regarding section of the new
debian developers reference, but I haven't yet done it :-)

Cheers.

-- 
Stefano Zacchiroli - undergraduate student of CS @ Univ. Bologna, Italy
zack@cs.unibo.it | ICQ# 33538863 | http://www.cs.unibo.it/~zacchiro
"I know you believe you understood what you think I said, but I am not
sure you realize that what you heard is not what I meant!" -- G.Romney

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]