Using primitives from dl libs

Using primitives from dl libs

[Alain Frisch]

Hello,

a few months ago, Nicolas George posted to this mailing list a patch
to allow using external (C) primitives without building a custom
runtime system (ocamlrun):
http://caml.inria.fr/archives/200004/msg00104.html

I am working on a similar solution. My approach is to add a new kind
of dynamically linked primitive declaration in the source code. For
instance:

external test : unit -> unit = "/home/frisch/caml/dl/test.so/ml_test"

ml_test is a C function defined in test.c (compiled to test.o then
converted to test.so with ld -shared).

Technically, I add a new section in the bytecode file listing the
primitives with a / in their name. The modified ocamlrun "dlopens"
the corresponding libraries at runtime. The patch (to the bytecode linker
and to ocamlrun) is really small.

Compared to Nicolas's solution, the advantage is to get rid of
the "primitive database" (a separate file indicating where to find
each primitive). Also, it is still possible to use statically linked
libraries.

Yet, I am not satisfied with my solution. Putting an explicit
filename for the library in the source file is ugly for obvious reasons.
Also, the decision to use dl for a specific library should be made during
the linking phase, or best, at runtime: if the bytecode interpreter
does not own a primitive, it searchs it in a shared lib.

The problem is: in which library should it search ?  I try to avoid
the primitive database. A solution is to provide this
information to ocamlrun via the command line (ocamlrun will
look up each primitive in every specified shared library).

Any suggestion about this design ?

-- 
  Alain Frisch

Re: Using primitives from dl libs

[Xavier Leroy]

> [Dynamic loading of C libraries implementing OCaml external functions]

> Yet, I am not satisfied with my solution. Putting an explicit
> filename for the library in the source file is ugly for obvious reasons.
> Also, the decision to use dl for a specific library should be made during
> the linking phase, or best, at runtime: if the bytecode interpreter
> does not own a primitive, it searchs it in a shared lib.

Yes, I agree it should work this way.

> The problem is: in which library should it search ?  I try to avoid
> the primitive database. A solution is to provide this
> information to ocamlrun via the command line (ocamlrun will
> look up each primitive in every specified shared library).

The solution I considered (but never had the fortitude to implement)
is to have an extra section in the bytecode executable listing the
shared libraries needed by the executable, i.e. all shared libraries
given to the OCaml linker with -cclib.

This section would contain short file names (e.g. libunix.so, not
/usr/local/lib/ocaml/libunix.so), and the bytecode interpreter would
search the corresponding .so files in a number of standard directories
(the directory containing the executable + the OCaml standard library
directory + the directories searched by the system dynamic loader, for
instance).

Then, the bytecode interpreter would dlopen() these libraries, and
dlsym() them all for each primitive.

I'm still not sure how to handle dynamic libraries that do not reside
in a standard directory.  We could have ocamlrun options or
environment variables (similar to LD_LIBRARY_PATH) to specify other
directories to search, but this isn't very nice.  Perhaps there should
be a file (in the stdlib directory) listing the directories to search,
a la /etc/ld.so.conf.

- Xavier Leroy