[Caml-list] Using OCaml from C

[Caml-list] Using OCaml from C

[Tom Ridge]

Dear List,

I want to use my ocaml code from a C program.

I have read the section "Interfacing C with OCaml" in the manual:

http://caml.inria.fr/pub/docs/manual-ocaml/manual033.html

The example in section 19.8 works fine. The compilation commands are
(in a Makefile; OCAMLPATH points to /home/tr61/.opam/4.00.1):

  ocamlc -custom -output-obj -o modcaml.o mod.ml
  ocamlc -c modwrap.c
  cp $(OCAMLPATH)/lib/ocaml/libcamlrun.a mod.a
  ar r mod.a modcaml.o modwrap.o
  cc -o prog main.c mod.a -lcurses -lm -ldl

(I had to add -lm and -ldl to get the example to work).

If my code makes use of the OCaml Unix library, I do:

cc -o prog main.c mod.a -lcurses -lm -ldl -L$(OCAMLPATH)/lib/ocaml -lunix

And if my code makes use of threads, I do:

cc -o prog main.c mod.a -lcurses -lm -ldl -L$(OCAMLPATH)/lib/ocaml
-lunix -lthreads -lpthread

So far, so good. The real code I want to use makes use of the Core
library. The compilation commands I finally got working are:

  ocamlfind ocamlc -package core -package str -linkpkg -thread -custom
-output-obj -o modcaml.o pqueue.ml mod.ml
  ocamlc -c modwrap.c
  cp $(OCAMLPATH)/lib/ocaml/libcamlrun.a mod.a
  ar r mod.a modcaml.o modwrap.o
  cc -pthread -Xlinker --allow-multiple-definition -o prog main.c
mod.a -lcurses -lm -ldl -lpthread  -L$(OCAMLPATH)/lib/ocaml -lunix
-lthreads -lbigarray -lcamlstr -lnums  -L$(OCAMLPATH)/lib/core_kernel
-lcore_kernel_stubs -L$(OCAMLPATH)/lib/core -lcore_stubs
-L$(OCAMLPATH)/lib/bin_prot -lbin_prot_stubs -lrt

For the last "cc" command, tt took quite a long time to figure out all
these flags and options (I do not know very much about linkers etc).
The errors I got were:

--

/home/tr61/.opam/4.00.1/lib/ocaml/libthreads.a(st_stubs_b.o): In
function `caml_thread_initialize':
st_stubs.c:(.text+0xed1): undefined reference to `pthread_atfork

fixed by adding -pthread after cc command

--

/home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
function `unix_clock_gettime':
unix_stubs.c:(.text+0x2776): undefined reference to `clock_gettime'

I had -lrt after -lpthread, but the error was still reported; when I
moved -lrt to end of command line things seemed to work; why?

--

/home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
function `unix_initgroups':
unix_stubs.c:(.text+0x3b90): multiple definition of `unix_initgroups'
/home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(initgroups.o):initgroups.c:(.text+0x0):
first defined here
/home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
function `unix_nice':
unix_stubs.c:(.text+0x46bc): multiple definition of `unix_nice'
/home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(nice.o):nice.c:(.text+0x0):
first defined here

fixed by adding -Xlinker --allow-multiple-definition to cc options

--

My real question is: what command *should* I be using to compile my
example? Or is the above more-or-less expected?

Thanks

Tom

RE: [Caml-list] Using OCaml from C

[David Allsopp]

Tom Ridge wrote:
> Dear List,
> 
> I want to use my ocaml code from a C program.
> 
> I have read the section "Interfacing C with OCaml" in the manual:
> 
> http://caml.inria.fr/pub/docs/manual-ocaml/manual033.html
> 
> The example in section 19.8 works fine. The compilation commands are (in a
> Makefile; OCAMLPATH points to /home/tr61/.opam/4.00.1):
> 
>   ocamlc -custom -output-obj -o modcaml.o mod.ml
>   ocamlc -c modwrap.c
>   cp $(OCAMLPATH)/lib/ocaml/libcamlrun.a mod.a
>   ar r mod.a modcaml.o modwrap.o
>   cc -o prog main.c mod.a -lcurses -lm -ldl
> 
> (I had to add -lm and -ldl to get the example to work).

The documentation is arguably deficient here - you can get those two extra switches from the output of ocamlc -config (cf bytecomp_c_libraries and native_c_libraries)

> If my code makes use of the OCaml Unix library, I do:
> 
> cc -o prog main.c mod.a -lcurses -lm -ldl -L$(OCAMLPATH)/lib/ocaml -lunix
> 
> And if my code makes use of threads, I do:
> 
> cc -o prog main.c mod.a -lcurses -lm -ldl -L$(OCAMLPATH)/lib/ocaml -lunix
> -lthreads -lpthread
> 
> So far, so good. The real code I want to use makes use of the Core
> library. The compilation commands I finally got working are:
> 
>   ocamlfind ocamlc -package core -package str -linkpkg -thread -custom -
> output-obj -o modcaml.o pqueue.ml mod.ml
>   ocamlc -c modwrap.c
>   cp $(OCAMLPATH)/lib/ocaml/libcamlrun.a mod.a
>   ar r mod.a modcaml.o modwrap.o
>   cc -pthread -Xlinker --allow-multiple-definition -o prog main.c mod.a -
> lcurses -lm -ldl -lpthread  -L$(OCAMLPATH)/lib/ocaml -lunix -lthreads -
> lbigarray -lcamlstr -lnums  -L$(OCAMLPATH)/lib/core_kernel -
> lcore_kernel_stubs -L$(OCAMLPATH)/lib/core -lcore_stubs -
> L$(OCAMLPATH)/lib/bin_prot -lbin_prot_stubs -lrt
> 
> For the last "cc" command, tt took quite a long time to figure out all
> these flags and options (I do not know very much about linkers etc).
> The errors I got were:

General tip for these: ocamlobjinfo library.cm[x]a will include in its output the required additional linking switches will save a few of these...

Adding -verbose (or using ocamlfind query) will allow you to locate the appropriate .cma or .cmxa for the findlib package in question.

I don't *think* that ocamlfind itself can help you with compilation in this direction (i.e. compiling OCaml to be used in C).

> --
> 
> /home/tr61/.opam/4.00.1/lib/ocaml/libthreads.a(st_stubs_b.o): In function
> `caml_thread_initialize':
> st_stubs.c:(.text+0xed1): undefined reference to `pthread_atfork
> 
> fixed by adding -pthread after cc command
> 
> --
> 
> /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> function `unix_clock_gettime':
> unix_stubs.c:(.text+0x2776): undefined reference to `clock_gettime'
> 
> I had -lrt after -lpthread, but the error was still reported; when I moved
> -lrt to end of command line things seemed to work; why?

Linker options for libraries should always be last (see gcc docs). I expect (but I'm sure a compiler expert can confirm/deny) that it's because the linker only pulls in symbols which it knows are needed (or at least it's allowed to do that - i.e. you need to specify foo.o requiring symbol bar before specifying -lfoo).

> --
> 
> /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> function `unix_initgroups':
> unix_stubs.c:(.text+0x3b90): multiple definition of `unix_initgroups'
> /home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(initgroups.o):initgroups.c:(.t
> ext+0x0):
> first defined here
> /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> function `unix_nice':
> unix_stubs.c:(.text+0x46bc): multiple definition of `unix_nice'
> /home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(nice.o):nice.c:(.text+0x0):
> first defined here
> 
> fixed by adding -Xlinker --allow-multiple-definition to cc options

Are you definitely allowed to use both Core and Unix in the same program (i.e. can you link on OCaml program that way?). Core is an stdlib replacement so I wouldn't expect to be allowed to use both standard library and core modules in the same program. If you are supposed to be able to do, then that sounds like a possible bug in core...

> --
> 
> My real question is: what command *should* I be using to compile my
> example? Or is the above more-or-less expected?

They look fine - I hope that the ocamlobjinfo tip allows for less trial-and-error in the future!


David 

RE: [Caml-list] Using OCaml from C

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 5204 bytes --]

Am Dienstag, den 10.09.2013, 12:41 +0000 schrieb David Allsopp:
> Tom Ridge wrote:
> > Dear List,
> > 
> > I want to use my ocaml code from a C program.
> > 
> > I have read the section "Interfacing C with OCaml" in the manual:
> > 
> > http://caml.inria.fr/pub/docs/manual-ocaml/manual033.html
> > 
> > The example in section 19.8 works fine. The compilation commands are (in a
> > Makefile; OCAMLPATH points to /home/tr61/.opam/4.00.1):
> > 
> >   ocamlc -custom -output-obj -o modcaml.o mod.ml
> >   ocamlc -c modwrap.c
> >   cp $(OCAMLPATH)/lib/ocaml/libcamlrun.a mod.a
> >   ar r mod.a modcaml.o modwrap.o
> >   cc -o prog main.c mod.a -lcurses -lm -ldl
> > 
> > (I had to add -lm and -ldl to get the example to work).
> 
> The documentation is arguably deficient here - you can get those two extra switches from the output of ocamlc -config (cf bytecomp_c_libraries and native_c_libraries)
> 
> > If my code makes use of the OCaml Unix library, I do:
> > 
> > cc -o prog main.c mod.a -lcurses -lm -ldl -L$(OCAMLPATH)/lib/ocaml -lunix
> > 
> > And if my code makes use of threads, I do:
> > 
> > cc -o prog main.c mod.a -lcurses -lm -ldl -L$(OCAMLPATH)/lib/ocaml -lunix
> > -lthreads -lpthread
> > 
> > So far, so good. The real code I want to use makes use of the Core
> > library. The compilation commands I finally got working are:
> > 
> >   ocamlfind ocamlc -package core -package str -linkpkg -thread -custom -
> > output-obj -o modcaml.o pqueue.ml mod.ml
> >   ocamlc -c modwrap.c
> >   cp $(OCAMLPATH)/lib/ocaml/libcamlrun.a mod.a
> >   ar r mod.a modcaml.o modwrap.o
> >   cc -pthread -Xlinker --allow-multiple-definition -o prog main.c mod.a -
> > lcurses -lm -ldl -lpthread  -L$(OCAMLPATH)/lib/ocaml -lunix -lthreads -
> > lbigarray -lcamlstr -lnums  -L$(OCAMLPATH)/lib/core_kernel -
> > lcore_kernel_stubs -L$(OCAMLPATH)/lib/core -lcore_stubs -
> > L$(OCAMLPATH)/lib/bin_prot -lbin_prot_stubs -lrt
> > 
> > For the last "cc" command, tt took quite a long time to figure out all
> > these flags and options (I do not know very much about linkers etc).
> > The errors I got were:
> 
> General tip for these: ocamlobjinfo library.cm[x]a will include in its output the required additional linking switches will save a few of these...
> 
> Adding -verbose (or using ocamlfind query) will allow you to locate the appropriate .cma or .cmxa for the findlib package in question.
> 
> I don't *think* that ocamlfind itself can help you with compilation in this direction (i.e. compiling OCaml to be used in C).

No, it cannot help here, because it cannot look inside the cma's to find
out the linker flags.

What I recommend here is to create an empty dummy.ml module, and do

ocamlfind ocamlc -verbose -package core,str -linkpkg -linkall -thread
-custom

which will print the final cc command ocamlc uses itself to link the
program. You can even intercept this command if you also pass
-cc ./my_own_script so that this script is called instead of cc. Just
keep all the -L and -l switches (and select other ones like -pthread).

The order of the -l switches matters: if you pass -lfoo -lbar to the
linker, library foo can use bar but not the other way round (note that
this is the opposite order you are used from ocaml).

> > /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> > function `unix_initgroups':
> > unix_stubs.c:(.text+0x3b90): multiple definition of `unix_initgroups'
> > /home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(initgroups.o):initgroups.c:(.t
> > ext+0x0):
> > first defined here
> > /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> > function `unix_nice':
> > unix_stubs.c:(.text+0x46bc): multiple definition of `unix_nice'
> > /home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(nice.o):nice.c:(.text+0x0):
> > first defined here
> > 
> > fixed by adding -Xlinker --allow-multiple-definition to cc options
> 
> Are you definitely allowed to use both Core and Unix in the same program (i.e. can you link on OCaml program that way?). Core is an stdlib replacement so I wouldn't expect to be allowed to use both standard library and core modules in the same program. If you are supposed to be able to do, then that sounds like a possible bug in core...

Looks like Core uses the same symbols as Unix. IMHO a bug in Core. Of
course you can use Core with other libraries (which will in turn use the
normal Unix module), because it's not a stdlib replacement, and you have
to open Core before you can see its definitions.

Gerd

> 
> > --
> > 
> > My real question is: what command *should* I be using to compile my
> > example? Or is the above more-or-less expected?
> 
> They look fine - I hope that the ocamlobjinfo tip allows for less trial-and-error in the future!
> 
> 
> David 
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] Using OCaml from C

[Stéphane Glondu]

Le 10/09/2013 14:21, Tom Ridge a écrit :
> /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> function `unix_clock_gettime':
> unix_stubs.c:(.text+0x2776): undefined reference to `clock_gettime'
> 
> I had -lrt after -lpthread, but the error was still reported; when I
> moved -lrt to end of command line things seemed to work; why?

The order of -l options matters, and a library must be put *after*
objects that use its symbols. System linkers have not always been strict
on the ordering, but they started a few years ago. Note that this is
usually the opposite of the order of OCaml objects.

In your case, clock_gettime is provided by -lrt, so -lrt must be after
-lcore_stubs.

> /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> function `unix_initgroups':
> unix_stubs.c:(.text+0x3b90): multiple definition of `unix_initgroups'
> /home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(initgroups.o):initgroups.c:(.text+0x0):
> first defined here
> /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> function `unix_nice':
> unix_stubs.c:(.text+0x46bc): multiple definition of `unix_nice'
> /home/tr61/.opam/4.00.1/lib/ocaml/libunix.a(nice.o):nice.c:(.text+0x0):
> first defined here
> 
> fixed by adding -Xlinker --allow-multiple-definition to cc options

Is -lunix really needed here? If so, I'd rather put it after
-lcore_stubs. Anyway, this doesn't look right and I would blame Core
here for using the same names as the standard Unix stubs.

> My real question is: what command *should* I be using to compile my
> example? Or is the above more-or-less expected?

Your calls to cp, ar and your use of -custom look suspicious. But it is
difficult to be more helpful without a concrete testcase.


Cheers,

-- 
Stéphane

RE: [Caml-list] Using OCaml from C

[David Allsopp]

Stéphane Glondu wrote:
> Le 10/09/2013 14:21, Tom Ridge a écrit :
> > /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
> > function `unix_clock_gettime':
> > unix_stubs.c:(.text+0x2776): undefined reference to `clock_gettime'
> >
> > I had -lrt after -lpthread, but the error was still reported; when I
> > moved -lrt to end of command line things seemed to work; why?

<snip>

> > My real question is: what command *should* I be using to compile my
> > example? Or is the above more-or-less expected?
> 
> Your calls to cp, ar and your use of -custom look suspicious. But it is
> difficult to be more helpful without a concrete testcase.

They're exactly as in the manual and are correct for compiling with ocamlc for static linking with C code. The cp and ar calls are necessary to set-up the static library (which should use libcamlrun.a as a starting point) - see #19.8 (the example in question), #19.7.4, #19.7.5 and #19.1.3 in the manual.


David 

Re: [Caml-list] Using OCaml from C

[Tom Ridge]

Dear List,

Thank you for all the replies. At least I feel that I am not doing
something *too* stupid. I will try to follow Gerd's suggestion and
inspect the final cc command from "ocamlfind ocamlc", and
cut-and-paste the relevant parts into the makefile.

Thanks

On 10 September 2013 14:30, David Allsopp <dra-news@metastack.com> wrote:
> Stéphane Glondu wrote:
>> Le 10/09/2013 14:21, Tom Ridge a écrit :
>> > /home/tr61/.opam/4.00.1/lib/core/libcore_stubs.a(unix_stubs.o): In
>> > function `unix_clock_gettime':
>> > unix_stubs.c:(.text+0x2776): undefined reference to `clock_gettime'
>> >
>> > I had -lrt after -lpthread, but the error was still reported; when I
>> > moved -lrt to end of command line things seemed to work; why?
>
> <snip>
>
>> > My real question is: what command *should* I be using to compile my
>> > example? Or is the above more-or-less expected?
>>
>> Your calls to cp, ar and your use of -custom look suspicious. But it is
>> difficult to be more helpful without a concrete testcase.
>
> They're exactly as in the manual and are correct for compiling with ocamlc for static linking with C code. The cp and ar calls are necessary to set-up the static library (which should use libcamlrun.a as a starting point) - see #19.8 (the example in question), #19.7.4, #19.7.5 and #19.1.3 in the manual.
>
>
> David
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Using OCaml from C

[Stéphane Glondu]

Le 10/09/2013 15:30, David Allsopp a écrit :
>> Your calls to cp, ar and your use of -custom look suspicious. But it is
>> difficult to be more helpful without a concrete testcase.
> 
> They're exactly as in the manual and are correct for compiling with ocamlc for static linking with C code. The cp and ar calls are necessary to set-up the static library (which should use libcamlrun.a as a starting point) - see #19.8 (the example in question), #19.7.4, #19.7.5 and #19.1.3 in the manual.

Well, that doesn't make them more right.

With the example of the manual, the following works and is much saner:

  ocamlc -output-obj -o modcaml.o mod.ml
  ocamlc -c modwrap.c
  ar rcs mod.a modcaml.o modwrap.o
  cc -o prog main.c mod.a -L`ocamlc -where` -lcamlrun -lm -ldl -lcurses

Copying libcamlrun.a around and fiddling with it also works and avoids
-L`ocamlc -where` -lcamlrun but it is just insane, as insane as
embedding the C standard library and runtime into third-party static
libraries.


Cheers,

-- 
Stéphane

Fwd: [Caml-list] Using OCaml from C

[Tom Ridge]

Just to make a final report:

Following Gerd and Stephane, I used ocamlfind -verbose to print the
options ocamlfind was passing to cc. After tidying, my Makefile now
looks like:

--

SHELL=bash

OCAMLPATH=/home/tr61/.opam/4.00.1/lib


all:
  ocamlfind ocamlc -package core,str -linkpkg -thread -output-obj -o
modcaml.o pqueue.ml mod.ml
  ocamlc -c modwrap.c
  rm -f mod.a
  ar rcs mod.a modcaml.o modwrap.o
  cc -o prog main.c mod.a -L$(OCAMLPATH)/bin_prot
-L$(OCAMLPATH)/variantslib -L$(OCAMLPATH)/num -L$(OCAMLPATH)/sexplib
-L$(OCAMLPATH)/fieldslib -L$(OCAMLPATH)/oUnit -L$(OCAMLPATH)/pa_ounit
-L$(OCAMLPATH)/res -L$(OCAMLPATH)/core_kernel -L$(OCAMLPATH)/core
-L$(OCAMLPATH)/ocaml/threads -L$(OCAMLPATH)/ocaml -lcamlstr
-lcore_stubs -lrt -lcore_kernel_stubs -lnums -lbin_prot_stubs
-lbigarray -lthreads -lunix -lpthread -lunix -lcamlrun
-I$(OCAMLPATH)/ocaml -lm  -ldl -lcurses -lpthread

--

which is better (e.g. no need for -pthread argument to cc, no need for
-Xlinker --allow-multiple-definition), although a bit verbose (but the
options were determined automatically, so this is a big time win).

Thanks



On 10 September 2013 15:21, Stéphane Glondu <steph@glondu.net> wrote:
> Le 10/09/2013 15:30, David Allsopp a écrit :
>>> Your calls to cp, ar and your use of -custom look suspicious. But it is
>>> difficult to be more helpful without a concrete testcase.
>>
>> They're exactly as in the manual and are correct for compiling with ocamlc for static linking with C code. The cp and ar calls are necessary to set-up the static library (which should use libcamlrun.a as a starting point) - see #19.8 (the example in question), #19.7.4, #19.7.5 and #19.1.3 in the manual.
>
> Well, that doesn't make them more right.
>
> With the example of the manual, the following works and is much saner:
>
>   ocamlc -output-obj -o modcaml.o mod.ml
>   ocamlc -c modwrap.c
>   ar rcs mod.a modcaml.o modwrap.o
>   cc -o prog main.c mod.a -L`ocamlc -where` -lcamlrun -lm -ldl -lcurses
>
> Copying libcamlrun.a around and fiddling with it also works and avoids
> -L`ocamlc -where` -lcamlrun but it is just insane, as insane as
> embedding the C standard library and runtime into third-party static
> libraries.
>
>
> Cheers,
>
> --
> Stéphane
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: Fwd: [Caml-list] Using OCaml from C

[Adrien Nader]

By the way,

ocamlmklib is a wrapper tool that will simplify some of these steps.

-- 
Adrien Nader

Re: Fwd: [Caml-list] Using OCaml from C

[Stéphane Glondu]

Le 10/09/2013 19:27, Adrien Nader a écrit :
> ocamlmklib is a wrapper tool that will simplify some of these steps.

"will"? Is there some plan to make it do that?

AFAIK, currently, it helps in doing mixed C/OCaml libraries usable from
OCaml, but not from C.

The process of building an OCaml library to be used from C looks like
making a regular executable from the OCaml point of view. ocamlmklib is
relevant here if you're exporting to C some OCaml code that uses itself
C code.


Cheers,

-- 
Stéphane