effect of -thread with ocamlc/ocamlopt -c

effect of -thread with ocamlc/ocamlopt -c

[Trevor Jim]

I have a question about the effect of -thread when used
with ocamlc/ocamlopt -c.

The manual states:

   All object files on the command line must also have been compiled
   with the -thread option

However, in compiling a large number of files I have not yet found a
case where -thread made a difference in the output of ocamlc or
ocamlopt (provided of course that module Thread is not used).  In
other words, the binary output is identical.

So, in general, can I take a .cmo or .cmx file that was NOT compiled
with -thread and link it with the threads library threads.cma or
threads.cmxa?

And, in general, can I take a .cmo or .cmx file that WAS compiled with
-thread but which does not use module Thread, and link it into a
program without the threads library?


Here's why this matters to me.  I have a large app (galax, with 300+
modules) that started out unthreaded.  I have added a threaded server,
which extends the app with one file that uses the Thread module.  So I
have a server that needs to be threaded, plus all the old programs
(galax compiler, etc.) that don't need to be threaded.  Linking in the
threads library causes the programs that don't use or need threads to
slow down by 2-30 times.

Since the program is so large, I would prefer not to have to compile
each module twice, once with threads and once without.

Also, it relies on a large number of libraries.  I don't want to have
to have two versions of each library.

But the manual seems to imply that I need to do that.

-Trevor

Re: [Caml-list] effect of -thread with ocamlc/ocamlopt -c

[Jonathan Roewen]

> The manual states:
>
>   All object files on the command line must also have been compiled
>   with the -thread option

In current compiler version, all the -thread option does is add the
include directory for threads to the include path automatically, and
set -nopervasives I think (so that it uses the version provided by the
thread library)..

Although, depending on the behaviour of compiler internals isn't
always a good thing ;-)

> So, in general, can I take a .cmo or .cmx file that was NOT compiled
> with -thread and link it with the threads library threads.cma or
> threads.cmxa?

Maybe (for example, the standard library doesn't have a special
compiled version for thread mode) -- just IO I'd be concerned about
wrt runtime behaviour.

> And, in general, can I take a .cmo or .cmx file that WAS compiled with
> -thread but which does not use module Thread, and link it into a
> program without the threads library?

Most likely yes. But again, you're depending on compiler internals.

> Since the program is so large, I would prefer not to have to compile
> each module twice, once with threads and once without.
>
> Also, it relies on a large number of libraries.  I don't want to have
> to have two versions of each library.
>
> But the manual seems to imply that I need to do that.

Without official comment from the ocaml team, expect upgrading to
newer ocaml releases to break things. Otherwise, should work for now.

Jonathan

Re: [Caml-list] effect of -thread with ocamlc/ocamlopt -c

[Jonathan Roewen]

Oh wait, I'm sorry. Yes, there is good reason: it uses a different
version of Pervasives internals.

Depending on when linking is done for each part, you may end up with
conflicting implementations of Pervasives in your final program: so
[IO] operations that can block will block whole program instead of
running thread blocking, and another thread running instead.

So maybe if it uses no IO, and no threading functions, it might be
okay -- there are no guarantees when you do these sorts of things (:

Jonathan

On 9/13/06, Trevor Jim <trevor@research.att.com> wrote:
> I have a question about the effect of -thread when used
> with ocamlc/ocamlopt -c.
>
> The manual states:
>
>   All object files on the command line must also have been compiled
>   with the -thread option
>
> However, in compiling a large number of files I have not yet found a
> case where -thread made a difference in the output of ocamlc or
> ocamlopt (provided of course that module Thread is not used).  In
> other words, the binary output is identical.
>
> So, in general, can I take a .cmo or .cmx file that was NOT compiled
> with -thread and link it with the threads library threads.cma or
> threads.cmxa?
>
> And, in general, can I take a .cmo or .cmx file that WAS compiled with
> -thread but which does not use module Thread, and link it into a
> program without the threads library?
>
>
> Here's why this matters to me.  I have a large app (galax, with 300+
> modules) that started out unthreaded.  I have added a threaded server,
> which extends the app with one file that uses the Thread module.  So I
> have a server that needs to be threaded, plus all the old programs
> (galax compiler, etc.) that don't need to be threaded.  Linking in the
> threads library causes the programs that don't use or need threads to
> slow down by 2-30 times.
>
> Since the program is so large, I would prefer not to have to compile
> each module twice, once with threads and once without.
>
> Also, it relies on a large number of libraries.  I don't want to have
> to have two versions of each library.
>
> But the manual seems to imply that I need to do that.
>
> -Trevor
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] effect of -thread with ocamlc/ocamlopt -c

[Trevor Jim]

Well, I'm still confused.  Consider that a bunch of the standard
libraries use Pervasives, but I can see that the standard library
is not compiled with -thread.  So clearly we are meant to use the
standard library compiled without -thread, with threads.cm[x]a.

So, I still don't understand when I can combine code compiled
without -thread with code compiled with -thread.

-Trevor


Jonathan Roewen wrote:
> Oh wait, I'm sorry. Yes, there is good reason: it uses a different
> version of Pervasives internals.
> 
> Depending on when linking is done for each part, you may end up with
> conflicting implementations of Pervasives in your final program: so
> [IO] operations that can block will block whole program instead of
> running thread blocking, and another thread running instead.
> 
> So maybe if it uses no IO, and no threading functions, it might be
> okay -- there are no guarantees when you do these sorts of things (:
> 
> Jonathan

Re: [Caml-list] effect of -thread with ocamlc/ocamlopt -c

[Jacques Garrigue]

The question is a bit different depending on whether you use system
threads or vmthreads. Note that if your platform supports system
threads, they are the ones used with the -thread option, while on
platforms without them -thread and -vmthread are equivalent.

For system threads, the standard library is unchanged, and you don't
need the -thread option for modules that don't use threads. So there
is no problem.

For vm threads, some modules from the standard library have different
implementations (namely, Pervasives, Marshal and Unix). So it is
essential that you use the -thread option to link with them. Note
however that the interfaces being identical, again you don't need
-thread for unrelated modules, so that in practice you shouldn't see
any difference with system threads. IIRC, at some point in the past
some modified functions were defined as external in the standard
library, making interfaces different, so that using -thread everywhere
was a strict requirement.

Jacques Garrigue

From: Trevor Jim <trevor@research.att.com>
> Well, I'm still confused.  Consider that a bunch of the standard
> libraries use Pervasives, but I can see that the standard library
> is not compiled with -thread.  So clearly we are meant to use the
> standard library compiled without -thread, with threads.cm[x]a.
> 
> So, I still don't understand when I can combine code compiled
> without -thread with code compiled with -thread.
> 
> -Trevor
> 
> Jonathan Roewen wrote:
> > Oh wait, I'm sorry. Yes, there is good reason: it uses a different
> > version of Pervasives internals.
> > 
> > Depending on when linking is done for each part, you may end up with
> > conflicting implementations of Pervasives in your final program: so
> > [IO] operations that can block will block whole program instead of
> > running thread blocking, and another thread running instead.
> > 
> > So maybe if it uses no IO, and no threading functions, it might be
> > okay -- there are no guarantees when you do these sorts of things (:
> > 
> > Jonathan

Re: [Caml-list] effect of -thread with ocamlc/ocamlopt -c

[Trevor Jim]

Thanks!

I hope that the manual can incorporate this clarification.

-Trevor

Jacques Garrigue wrote:
> The question is a bit different depending on whether you use system
> threads or vmthreads. Note that if your platform supports system
> threads, they are the ones used with the -thread option, while on
> platforms without them -thread and -vmthread are equivalent.
> 
> For system threads, the standard library is unchanged, and you don't
> need the -thread option for modules that don't use threads. So there
> is no problem.
> 
> For vm threads, some modules from the standard library have different
> implementations (namely, Pervasives, Marshal and Unix). So it is
> essential that you use the -thread option to link with them. Note
> however that the interfaces being identical, again you don't need
> -thread for unrelated modules, so that in practice you shouldn't see
> any difference with system threads. IIRC, at some point in the past
> some modified functions were defined as external in the standard
> library, making interfaces different, so that using -thread everywhere
> was a strict requirement.
> 
> Jacques Garrigue

Re: [Caml-list] effect of -thread with ocamlc/ocamlopt -c

[Xavier Leroy]

Jacques Garrigue wrote:
> The question is a bit different depending on whether you use system
> threads or vmthreads. [...]

Jacques' explanation is correct: there was a time when the requirement
listed in the manual ("all modules of a multithreaded program must be
compiled with ocamlc -thread ") made sense, because some standard
library modules had different *interfaces* between the non-threaded
and threaded versions.  This is no longer the case: with system
threads, those modules have the same interfaces and the same
implementations, and even with bytecode threads they have different
implementations but identical interfaces.

So, you can safely ignore this sentence in the manual.  The manual
will be updated for the next major release.

- Xavier Leroy