C libs from Ocaml libs

C libs from Ocaml libs

[viktor tron]

[-- Attachment #1: Type: text/plain, Size: 4416 bytes --]

Dear list,

Let me share with you my frustration over compiling a c library out of an
ocaml project. [apologies for long post]

This is now my third project where an ocaml implementation needs a c binding
and I keep on struggling
with the right build environment.

While it is very well documented how to compile and build applications with
ocaml bindings to
c implementations, the opposite way is a neglected topic.
sections 18.7-8 of the manual
http://caml.inria.fr/pub/docs/manual-ocaml/manual032.html
plus a couple of pages of the o-reilly book is devoted to this topic.
The sporadic info found on mailing lists is not enough so I need your help.

The well known abstract problem is the following:

foo.ml: the ocaml routine root (responsible for foo.cmxa)
foo_export.ml the C API (register callback, exceptions, etc)
foo_stub.c the C bindings referring to the callbacks, containing init code
(foo_wrapper.ml the ocaml wrapper to foo)

clearly what I want is the following:

* foo.cmxa ocaml lib with ocaml entrypoints
* (foo: standalone executable from foo.cmxa and foo_wrapper.ml)
* libfoo: static and dynamic C library out of foo.cmxa, foo_export and
foo_stub

what do I do? now more or less according to the manual:

ocamlc -a <all files> fib.ml
ocamlc -custom -output-obj -o fib_caml.o fib.cma fib_export.ml
ocamlc -c fib_stub.c
cp /usr/local/lib/ocaml/libcamlrun.a fib.a
ar r fib.a fib_caml.o fib_stub.o
ranlib fib.a

or

ocamlc -a <all files> foo.ml
ocamlc -custom -output-obj -o foo_caml.o foo.cma foo_export.ml
ocamlc -c foo_stub.c
libtool -o libfoo.a foo_caml.o foo_stub.o /usr/local/lib/ocaml/libcamlrun.a

and then link the c program with foo

* I know that building on several platforms is always a challange but this
is such a simple routine.
if I have to use random command lines in my Makefile, I run into an autoconf
hell.
is their a truly plantform independent routine to do this?

* can someone explain to me how the how exactly .a file created with .cmxa
differ from an archive?
is it the ocaml sturtup code?

* why does the bytecode need the curses library when linking and why the
native doesn't?

* why does ocamlc have a -custom option and why does ocamlopt does not? do I
need the -custom options at all here?

* what do I do if I don't want to compile the ocaml runtime into my lib,
since they might use several
of my modules so just link at the end with the runtime lib
* what if I want to share ocaml code for the same reason?

and most importantly:

* how do I create dynamic libs, dlls for windows?

In general, I found not much help from the existing build tools:

* OCamlMake does not support creation of c libraries

* omake is serious overkill (after hours of reading through the manual, I
still don't know how the gtk example is relevant, plus omake creates an
extra problem for portability, please correct me if I am wrong)

* ocamlbuild? I have problems with this piece of software. I found it
frustatingly underdocumented and buggy.
I read the manual, help and the -documentation, and still don't understand
anything.
The examples on the wiki page
http://brion.inria.fr/gallium/index.php/C_stub_functions
give 100 lines of ocaml code for a similar problem?
- I still don't know the most basic thing, how to build an output-obj o
file.
- I found no way to convince ocamlbuild to use the local library (in this
example we build
ocamlbuild foo.cmxa
and then would simply want
ocamlopt -o foo foo.cmxa foo_wrapper.ml
but 'ocamlbuild foo_wrapper.native' rebuilds it all the way.

* I even tried to get inpiration from rocaml (a nice ruby binding for ocaml,
which is mediated by a c, and all build is automated, but
it was too complicated to delineate the ruby-relevant things from the
c-ocaml interface)

* why are the c object files end in .o irrespective of whether they are byte
or native?
this makes it impossible to write rules on them?
In fact 'ocamlbuild foo.o' will build both and it overwrites the targets of
the first round!!!
(since it matches both cmo & o and cmx & o rules?)


I know it is a lot of questions, but any help or comment is appreciated.

Something constructive:
Does the community not feel that we should somehow support this export much
more within our existing build tools and documentation? If something
reassuring comes out of this discussion I promise I write up a tutorial.
(At least I understand how to write the code)

Thanks very much.

viktor

[-- Attachment #2: Type: text/html, Size: 5517 bytes --]

Re: [Caml-list] C libs from Ocaml libs

[Alain Frisch]

viktor tron wrote:
> This is now my third project where an ocaml implementation needs a c 
> binding and I keep on struggling
> with the right build environment.

What you want is a main program in C, with some services provided by 
OCaml code, right?

> what do I do? now more or less according to the manual:
> 
> ocamlc -a <all files> fib.ml <http://fib.ml>
> ocamlc -custom -output-obj -o fib_caml.o fib.cma fib_export.ml
> ocamlc -c fib_stub.c
> cp /usr/local/lib/ocaml/libcamlrun.a fib.a
> ar r fib.a fib_caml.o fib_stub.o
> ranlib fib.a
> 
> or
> 
> ocamlc -a <all files> foo.ml <http://foo.ml>
> ocamlc -custom -output-obj -o foo_caml.o foo.cma foo_export.ml
> ocamlc -c foo_stub.c
> libtool -o libfoo.a foo_caml.o foo_stub.o /usr/local/lib/ocaml/libcamlrun.a
> 
> and then link the c program with foo

Note that you don't need -custom here (-output-obj implies -custom).

> * I know that building on several platforms is always a challange but 
> this is such a simple routine.
> if I have to use random command lines in my Makefile, I run into an 
> autoconf hell.
> is their a truly plantform independent routine to do this?

Well, if your main application is in C, I guess your build system will 
need to know somewhat on which platform it runs.

Anyway, in the current development branch on OCaml, it is possible to 
build a dynamic library that contains the OCaml runtime + arbitrary 
OCaml and C code, in a platform-independent way. This is similar to 
linking a main program, but the result is linked as a DLL.

In your case:

ocamlc -output-obj -o fib.so fib_stub.c fib.cma fib_export.ml

or:

ocamlopt -output-obj -o fib.so fib_stub.c fib.cmxa fib_export.ml

(You can also compile the .c and .ml files separately, of course.)

Cf the test2 and testopt2 targets in:
http://camlcvs.inria.fr/cgi-bin/cvsweb/~checkout~/ocaml/test/outputobj/Makefile

(This is an example of a main program in C#, but the same technique also 
  works for a main program in C. In addition, the example uses the 
Dynlink module, but this is orthogonal to your questions.)

> * can someone explain to me how the how exactly .a file created with 
> .cmxa differ from an archive?
> is it the ocaml sturtup code?

The .a file is an archive. It contains the native code produced by 
ocamlopt from OCaml sources. The .cmxa only contains "meta-information" 
about the library.

> * why does ocamlc have a -custom option and why does ocamlopt does not? 
> do I need the -custom options at all here?

-custom means: build a custom version of the OCaml runtime system with 
custom C libraries and objects linked in. This is used when one doesn't 
want to (or cannot) use dynamically link C libraries (such as 
dllunix.so) is not an option. Without -custom, a main program produced 
by ocamlc only contains OCaml bytecode (and a reference to ocamlrun).

Since executables produced by ocamlopt are native executable, it is 
possible to link C libraries and objects directly. There is no need for 
a special -custom option.

In your case, you don't need -custom (see above).

> * what do I do if I don't want to compile the ocaml runtime into my lib, 
> since they might use several
> of my modules so just link at the end with the runtime lib

Well, if you don't want to compile the ocaml runtime into your lib, just 
  don't do it. Simply create an archive with the .o produced by 
-output-obj and your C code. Or don't create an archive at all and link 
everything directly into your final executable.

See Section 18.7.5 of the OCaml manual.

> * how do I create dynamic libs, dlls for windows?

What do you want to put in your dynamic libs? If you want to create a 
"stand-alone" dll with the OCaml runtime + arbitrary OCaml and C code, 
the new behavior of "-output-obj -o XXX.{so,dll}" is/will be what you want.

> * omake is serious overkill (after hours of reading through the manual, 
> I still don't know how the gtk example is relevant, plus omake creates 
> an extra problem for portability, please correct me if I am wrong)

Quite the contrary, I would say. Portability seems like a very 
compelling reason to adopt omake. It works very nicely under Unix and 
Windows.

> * why are the c object files end in .o irrespective of whether they are 
> byte or native?

Files produced by -output-obj are native objects, even if produced by 
ocamlc (in that case, they contain the bytecode as data, but they really 
are native objects). You can choose the base name you want for the 
output of -output-obj, so it is up to you (or your build system) to 
avoid conflicts.


-- Alain

Re: [Caml-list] C libs from Ocaml libs

[Matthieu Dubuget]

2007/11/21, Alain Frisch <alain@frisch.fr>:
> > * how do I create dynamic libs, dlls for windows?
>
> What do you want to put in your dynamic libs? If you want to create a
> "stand-alone" dll with the OCaml runtime + arbitrary OCaml and C code,
> the new behavior of "-output-obj -o XXX.{so,dll}" is/will be what you want.
>

Waou! I did not noticed this!
Thanks for this information Alain.
Since when is it possible?
I can't find anything about it in the doc.
Except if you take xxx.dll as an executable.

Ĝis!

Matt

Re: [Caml-list] C libs from Ocaml libs

[Alain Frisch]

Matthieu Dubuget wrote:
> 2007/11/21, Alain Frisch <alain@frisch.fr>:
>>> * how do I create dynamic libs, dlls for windows?
>> What do you want to put in your dynamic libs? If you want to create a
>> "stand-alone" dll with the OCaml runtime + arbitrary OCaml and C code,
>> the new behavior of "-output-obj -o XXX.{so,dll}" is/will be what you want.
>>
> 
> Waou! I did not noticed this!
> Thanks for this information Alain.
> Since when is it possible?

This is part of the set of changes that will come with the features 
previously found on the natdynlink branch (Dynlink in native code, 
simplified treatment of DLLs under Windows, unification of Makefiles 
between Unix and Windows,...). I guess this will be available in OCaml 
3.11 (to be expected not too soon); currently, you can find it on the 
main branch of OCaml's CVS.

-- Alain

Re: [Caml-list] C libs from Ocaml libs

[Jon Harrop]

On Wednesday 21 November 2007 12:05, Alain Frisch wrote:
> This is part of the set of changes that will come with the features
> previously found on the natdynlink branch (Dynlink in native code,
> simplified treatment of DLLs under Windows, unification of Makefiles
> between Unix and Windows,...). I guess this will be available in OCaml
> 3.11 (to be expected not too soon)...

Does this include your rather awesome ocamlnat native-code top-level?

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] C libs from Ocaml libs

[viktor tron]

[-- Attachment #1: Type: text/plain, Size: 3643 bytes --]

On 21/11/2007, Alain Frisch <alain@frisch.fr> wrote:
>
> viktor tron wrote:
> > This is now my third project where an ocaml implementation needs a c
> > binding and I keep on struggling
> > with the right build environment.
>
> What you want is a main program in C, with some services provided by
> OCaml code, right?


Alain, thank you very much for taking your time and clarify things for me.
Well, the MAIN program can be anything that can handle a correct C-lib.
Actually, I had projects in C++, C, Ruby that called my lib.
In fact, my next assignment is a .NET integration so your C# example will be
a lifesaver for me I guess.

Anyway, in the current development branch on OCaml, it is possible to
> build a dynamic library that contains the OCaml runtime + arbitrary
> OCaml and C code, in a platform-independent way. This is similar to
> linking a main program, but the result is linked as a DLL.


This is super! how is this on MacOS, I recall one of your comment
earlier (on this list?)
that it doesn't work or something.


> Cf the test2 and testopt2 targets in:
>
> http://camlcvs.inria.fr/cgi-bin/cvsweb/~checkout~/ocaml/test/outputobj/Makefile



(This is an example of a main program in C#, but the same technique also
>   works for a main program in C. In addition, the example uses the
> Dynlink module, but this is orthogonal to your questions.)


Very cool, I will have a thourough look.

> * what do I do if I don't want to compile the ocaml runtime into my lib,
> > since they might use several
> > of my modules so just link at the end with the runtime lib
>
> Well, if you don't want to compile the ocaml runtime into your lib, just
>   don't do it. Simply create an archive with the .o produced by
> -output-obj and your C code. Or don't create an archive at all and link
> everything directly into your final executable.
>
> See Section 18.7.5 of the OCaml manual.


well, I tried.
ar -rs foo_caml.o foo_stub.o

but then I get linking errors:

gcc -o foo_test.native foo_test.c -L. -lchainfreq_native -L/sw/lib/ocaml
-lasmrun
/usr/bin/ld: Undefined symbols:
_caml_apply2
...
lots of other symbols

I guess I am missing something pretty trivial here, so apologies to the
savvy.


> * how do I create dynamic libs, dlls for windows?
>
> What do you want to put in your dynamic libs? If you want to create a
> "stand-alone" dll with the OCaml runtime + arbitrary OCaml and C code,
> the new behavior of "-output-obj -o XXX.{so,dll}" is/will be what you
> want.


and not standalone ones?

> * omake is serious overkill (after hours of reading through the manual,
> > I still don't know how the gtk example is relevant, plus omake creates
> > an extra problem for portability, please correct me if I am wrong)


Quite the contrary, I would say. Portability seems like a very
> compelling reason to adopt omake. It works very nicely under Unix and
> Windows.


yes, in general omake is super appealing, but (a) every dependency my
software relies on
discourages/angers users. It is already a pain for people to have the ocaml
compiler and a shell
already. I know this is another
world... (b) I still don't know how it would help automate my problem
yet, as I said the gtk example is a serious overkill.

On a sidenote to this issue, the rocaml package uses a camlp4 extention to
create callback registration and automates the creation of a stub file. (And
as far as I understood, omake does something similar).
Since most of my callbacks use a pretty simple API (meaning I don't have to
deal with conversion of esoteric ocaml types), the whole process could
easily be automated and integrated in my build template.

Viktor

[-- Attachment #2: Type: text/html, Size: 6316 bytes --]

Re: [Caml-list] C libs from Ocaml libs

[Alain Frisch]

viktor tron wrote:
> This is super! how is this on MacOS, I recall one of your comment earlier (on this list?)
> that it doesn't work or something. 

Everything should work fine under Mac OS X x86. For PowerPC, I believe 
that "ocamlc -output-obj -o XXX.so" should be ok but that "ocamlopt 
-output-obj -o XXX.so" does not work.

> well, I tried.
> ar -rs foo_caml.o foo_stub.o
 > gcc -o foo_test.native foo_test.c -L. -lchainfreq_native -L/sw/lib/ocaml

What is the name of the library you want to produce?

>      > * how do I create dynamic libs, dlls for windows?
> 
>     What do you want to put in your dynamic libs? If you want to create a
>     "stand-alone" dll with the OCaml runtime + arbitrary OCaml and C code,
>     the new behavior of "-output-obj -o XXX.{so,dll}" is/will be what
>     you want. 
> 
> 
> and not standalone ones?

So, you want the OCaml code, but not the OCaml runtime?  You can either 
use the -shared option to produce a .cmxs plugin (that must be 
explicitly loaded by the main program, using the OCaml Dynlink module), 
or -output-obj to produce a .o containing the OCaml code and the startup 
code for the OCaml side of your application.

-- Alain

Re: [Caml-list] C libs from Ocaml libs

[viktor tron]

[-- Attachment #1: Type: text/plain, Size: 2484 bytes --]

On 21/11/2007, Alain Frisch <alain@frisch.fr> wrote:
>
> viktor tron wrote:
> > This is super! how is this on MacOS, I recall one of your comment
> earlier (on this list?)
> > that it doesn't work or something.
>
> Everything should work fine under Mac OS X x86. For PowerPC, I believe
> that "ocamlc -output-obj -o XXX.so" should be ok but that "ocamlopt
> -output-obj -o XXX.so" does not work.
>
> > well, I tried.
> > ar -rs foo_caml.o foo_stub.o
> > gcc -o foo_test.native foo_test.c -L. -lchainfreq_native -L/sw/lib/ocaml
>
>
> What is the name of the library you want to produce?


sorry, let me be very explicit then:

0)
I believed that the following
creates the c object containing:
(a) main ocaml implementation of foo (b) export API (c) C binding to API (d)
ocaml startup code

$ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml

now I create a lib

$ ar rs libfoo.a foo_caml.o

and happy ever after. Nope.
No matter how I link it with a main c test, I get undefined symbols
for startup code


$ gcc -o foo_test foo_test.c -L. -lfoo -lasmrun

In fact, the following variants don't work:

1)
$ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
$ ar -rs libfoo.a /sw/lib/ocaml/libasmrun.a foo_caml.o
$ gcc -o foo_test foo_test.c -L. -lfoo

2)
$ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
$ cp /sw/lib/ocaml/libasmrun.a libfoo.a
$ ar -rs libfoo.a foo_caml.o
$ gcc -o foo_test foo_test.c -L. -lfoo

so it seems I have to compile foo_stub.o separately...

but even if I do:

3)
$ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
$ ocamlopt foo_stub.c
$ ar -rs libfoo.a foo_caml.o foo_stub.o
$ gcc -o foo_test foo_test.c -L. -lfoo -L/sw/lib/ocaml -lasmrun

4)
$ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
$ ocamlopt foo_stub.c
$ ar -rs libfoo.a /sw/lib/ocaml/libasmrun.a foo_caml.o
$ gcc -o foo_test foo_test.c -L. -lfoo

both fail, the only thing that works is when
(a) I compile stub and caml objects separately AND
(b) use copy to bootstrap the runtime

$ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
$ ocamlopt foo_stub.c
$ cp /sw/lib/ocaml/libasmrun.a libfoo.a
$ ar -rs libfoo.a foo_caml.o foo_stub.o
$ gcc -o foo_test foo_test.c -L. -lfoo

In fact ocamlopt does pretty enigmatic things in the background and ar is an
entire mystery with this
copy thing.
If anyone can make me understand the above, I'd be forever grateful.

thanks
Viktor

[-- Attachment #2: Type: text/html, Size: 3388 bytes --]

Re: [Caml-list] C libs from Ocaml libs

[Alain Frisch]

viktor tron wrote:
> sorry, let me be very explicit then:
> 
> 0)
> I believed that the following
> creates the c object containing:
> (a) main ocaml implementation of foo (b) export API (c) C binding to API 
> (d) ocaml startup code
> 
> $ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml

You are correct.

> now I create a lib
> 
> $ ar rs libfoo.a foo_caml.o
> 
> and happy ever after. Nope.
> No matter how I link it with a main c test, I get undefined symbols for startup code 
> 
> 
> $ gcc -o foo_test foo_test.c -L. -lfoo -lasmrun

What happens if you pass foo_caml.o instead of -lfoo?

Can you send me (maybe off-list) a full set of files to reproduce the 
problem?

> In fact, the following variants don't work:
> 
> 1)
> $ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
> $ ar -rs libfoo.a /sw/lib/ocaml/libasmrun.a foo_caml.o
> $ gcc -o foo_test foo_test.c -L. -lfoo

I don't think that ar supports merging libraries like that. What can be 
done is to use "ld -r" (partial linking) to produce a single object file
from libasmrun.a and foo_caml.o and then put this file in a library (or 
use it directly).

> 2)
> $ ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_stub.c foo_export.ml
> $ cp /sw/lib/ocaml/libasmrun.a libfoo.a
> $ ar -rs libfoo.a foo_caml.o

Ok, that's a hack to merge one library, plus more object files into a 
library. I know it is mentioned in the OCaml manual. Personally, I would 
not recommend to do that (you'll be stuck if you need to include e.g. 
libunix.a).

> In fact ocamlopt does pretty enigmatic things in the background

Use the "-verbose" option to see what's going on.

> and ar 
> is an entire mystery with this
> copy thing.

Use "ar t" to see what's in a library.

-- Alain

Re: [Caml-list] C libs from Ocaml libs

[Alain Frisch]

viktor tron wrote:
> If anyone can make me understand the above, I'd be forever grateful.

Here is the more direct way to achive what you want:

===========================================================
ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_export.ml
ocamlopt -c foo_stub.c
gcc -o foo_test foo_test.c foo_stub.o foo_caml.o \
    $(OCAMLLIB)/libasmrun.a -lm -ldl
===========================================================


Note that "ocamlopt -output-obj" uses the so-called partial linking 
feature (ld -r) to produce a single object file that contains:
- the startup code
- all the native OCaml code (including the code of the libraries, like 
  foo.a and stdlib.a)

But *not* the extra C objects and libraries (-ccopt, *.c/*.o/*.a 
arguments). (Thanks to Rabih for this off-line reminder.) I don't know 
whether this qualifies as a bug or as a feature.

Here is a variant where we pack foo_caml.o and foo_stub.o together in a 
single object using the partial linker manually:

===========================================================
ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_export.ml
ocamlopt -c foo_stub.c
ld -r -o libfoo.o foo_stub.o foo_caml.o	
gcc -c foo_test.c
gcc -o foo_test foo_test.c libfoo.o $(OCAMLLIB)/libasmrun.a -lm -ldl
===========================================================

We can do more and also put the libasmrun.a library in the same object:

===========================================================
ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_export.ml
ocamlopt -c foo_stub.c
ld -r -o libfoo.o foo_stub.o foo_caml.o	$(OCAMLLIB)/libasmrun.a
gcc -c foo_test.c
gcc -o foo_test foo_test.c libfoo.o -ldl -lm
===========================================================

One could try to put foo_stub.o and foo_caml.o together in a library:

===========================================================
ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_export.ml
ocamlopt -c foo_stub.c
ar r libfoo.a foo_stub.o foo_caml.o	
gcc -c foo_test.c
gcc -o foo_test foo_test.c libfoo.a $(OCAMLLIB)/libasmrun.a -lm -ldl
===========================================================

This does not work because the two libraries libfoo.a and libasmrun.a 
depend on symbols defined in the other library. This can be addressed 
with the linker options --start-group/--end-group:

===========================================================
ocamlopt -output-obj -o foo_caml.o foo.cmxa foo_export.ml
ocamlopt -c foo_stub.c
ar r libfoo.a foo_stub.o foo_caml.o	
gcc -c foo_test.c
gcc -o foo_test foo_test.c \
  -Wl,--start-group libfoo.a $(OCAMLLIB)/libasmrun.a -Wl,--end-group \
  -lm -ldl
===========================================================


In the current development version of OCaml, it is also possible to go 
through a shared library:

===========================================================
ocamlopt -output-obj -o foo_caml.so foo.cmxa foo_export.ml foo_stub.c
gcc -c foo_test.c
gcc -o foo_test foo_test.c foo_caml.so -Wl,-R.
===========================================================

When -output-obj is used to produce a shared library, it will put extra 
C objects and options (here, foo_stub.o, libasmrun.a -lm, -ldl) in the 
result.

Feel free to ask more questions (and to write a tutorial on this issues)!


Alain