Embedding the ocaml runtime in a shared library on amd64/x86_64

Embedding the ocaml runtime in a shared library on amd64/x86_64

[Joost Yervante Damad]

Hello,

while trying to get the ocaml runtime embedded in a shared library, I ran into issues similar to [1]. 
Basically there are quite some symbols that have a R_X86_64_32 relocation in libasmrun.a.
I worked around that by rebuilding libasmrun.a with -fPIC, however there still is the file amd64.S which gives errors like this:

usr/bin/ld: /usr/lib/ocaml/libasmrun.a(amd64.o): relocation R_X86_64_PC32 against symbol `caml_last_return_address' can not be used when making a shared 
object; recompile with -fPIC

For that I found the following workaround: add the -Bsymbolic linker option to the command that creates the shared library, something like this:

ocamlopt -ccopt -Wl,-Bsymbolic -verbose -o libmod.so -ccopt -shared mod.cmx modwrap.o

Then my C application links with libmod.so and runs.
It should be possible to adapt amd64.S to generate the correct relocation entries by adding @PLT to some of the symbols, something like this:

movq    %r15, G(caml_young_ptr)@PLT(%rip)

However while it removed the need to use -Bsymbolic when linking, the resulting library seems to be faulty because the application segfaults.

I think compiling with -fPIC by default on the amd64/x86_64 arch would be a good start to making it more usable for embedding.
Also it would be very interesting if someone with good knowledge of ocaml internals could fix amd64.S to be position independent.

Greetings, Joost Damad

[1] http://groups.google.com/group/fa.caml/browse_thread/thread/d734c80bc3fc5a1d/395cd691e481ceb7

-- 
Joost Yervante Damad - http://damad.be/joost/

Re: [Caml-list] Embedding the ocaml runtime in a shared library on amd64/x86_64

[Richard Jones]

On Wed, Dec 02, 2009 at 01:33:11PM +0100, Joost Yervante Damad wrote:
> I think compiling with -fPIC by default on the amd64/x86_64 arch
> would be a good start to making it more usable for embedding.

I think this is also a very good idea.  We went through this
already with libcamlrun.a:

http://caml.inria.fr/mantis/view.php?id=3866

So I think you should open a similar bug.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Embedding the ocaml runtime in a shared library on amd64/x86_64

[Stefano Zacchiroli]

On Wed, Dec 02, 2009 at 04:51:16PM +0000, Richard Jones wrote:
> On Wed, Dec 02, 2009 at 01:33:11PM +0100, Joost Yervante Damad wrote:
> > I think compiling with -fPIC by default on the amd64/x86_64 arch
> > would be a good start to making it more usable for embedding.
> I think this is also a very good idea.

Seconded.

> We went through this already with libcamlrun.a:
> http://caml.inria.fr/mantis/view.php?id=3866

Just to clarify a bit more for who did not follow that bug: what we did
there was not precisely what Joost asked for (i.e. build libasmrun.a
with -fPIC). Rather the patch offers two different libraries:
libcamlrun.a (no -fPIC) and libcamlrun_shared.so (-fPIC). The reason was
a performance penalty in using the -fPIC version.

Both in Debian and Fedora package you now have both libraries.

That said, I share Rich's suggestion: please file a bug (with patch
would be even better :)), that enable building 2 different versions of
libcamlrun, one built with -fPIC, the other as it is now.

Cheers.

-- 
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime

Re: [Caml-list] Embedding the ocaml runtime in a shared library on amd64/x86_64

[Joost Yervante Damad]

On Wednesday 02 December 2009 23:37:51 Stefano Zacchiroli wrote:
> On Wed, Dec 02, 2009 at 04:51:16PM +0000, Richard Jones wrote:
> > On Wed, Dec 02, 2009 at 01:33:11PM +0100, Joost Yervante Damad wrote:
> > > I think compiling with -fPIC by default on the amd64/x86_64 arch
> > > would be a good start to making it more usable for embedding.
> >
> > I think this is also a very good idea.
> 
> Seconded.
> 
> > We went through this already with libcamlrun.a:
> > http://caml.inria.fr/mantis/view.php?id=3866
> 
> Just to clarify a bit more for who did not follow that bug: what we did
> there was not precisely what Joost asked for (i.e. build libasmrun.a
> with -fPIC). Rather the patch offers two different libraries:
> libcamlrun.a (no -fPIC) and libcamlrun_shared.so (-fPIC). The reason was
> a performance penalty in using the -fPIC version.

Reading up on this, I don't think using -fPIC on amd64 has a performance 
penalty since it has the %rip register. Also it appears that gcc by default 
uses indirect addressing via %rip in the generated asm.

> Both in Debian and Fedora package you now have both libraries.
> 
> That said, I share Rich's suggestion: please file a bug (with patch
> would be even better :)), that enable building 2 different versions of
> libcamlrun, one built with -fPIC, the other as it is now.

I'm first working out the details for a larger application to make sure that 
also works.
I'll file a bug when I have a fully working solution.

greetings, Joost Damad

-- 
Joost Yervante Damad - http://damad.be/joost/

Re: [Caml-list] Embedding the ocaml runtime in a shared library on amd64/x86_64

[Alain Frisch]

Joost Yervante Damad wrote:
> Reading up on this, I don't think using -fPIC on amd64 has a performance 
> penalty since it has the %rip register. Also it appears that gcc by default 
> uses indirect addressing via %rip in the generated asm.

I did some benchmarks when switching the ocamlopt AMD64 code generator 
to generate PIC code by default (in order to support Dynlink on native 
code): the slowdown was indeed very small (in the order of a few percent 
for micro benchmarks). Modern processors seem very good for optimizing 
indirect calls.


Alain

Re: [Caml-list] Embedding the ocaml runtime in a shared library on amd64/x86_64

[Stefano Zacchiroli]

On Thu, Dec 03, 2009 at 11:56:59AM +0100, Joost Yervante Damad wrote:
> Reading up on this, I don't think using -fPIC on amd64 has a performance 
> penalty since it has the %rip register. Also it appears that gcc by default 
> uses indirect addressing via %rip in the generated asm.

I duly notice that OCaml is not only compiler on "modern processors".
So, a solution that does not offer the non-fPIC solution might be more
than appropriate on amd64 processors, but maybe not as much appropriate
on other architectures (Debian, for instance, support about 10 other
architectures in addition to amd64; even if we restrict to architecture
having ocamlopt, there is more than plain amd64).

Note: I haven't checked whether there is a difference on all such
architectures for what concerns libasmrun, but there were difference for
such archs for libcamlrun.

Given the negligible cost (both in term of patch preparation and of
solution complexity) of supporting the build of both kind of libraries,
I don't see why we should simply switch to the -fPIC library, trashing
the other.

Thanks for your work on this!
Cheers.

-- 
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime

Re: [Caml-list] Embedding the ocaml runtime in a shared library on amd64/x86_64 [update]

[Joost Yervante Damad]

Hello all,

a short update on my efforts to embed the ocaml runtime on amd64:

static linking the ocaml runtime works rather fine like this:

1. -fPIC

Make sure all files in asmrun are compiled with -fPIC by adapting the Makefile.

2) asmrun/amd64.S

    use an adapted amd64.S where 3 symbols are marked with @PLT:
    caml_program, caml_apply2, caml_apply3 like this:
        leaq    G(caml_program)@PLT(%rip), %r12
        leaq    G(caml_apply2)@PLT(%rip), %r12  /* code pointer */
        leaq    G(caml_apply3)@PLT(%rip), %r12  /* code pointer */

3) link the shared library

    # ocaml test code:
    ocamlopt -verbose -g -c mod.ml
    # wrapper code in C:
    ocamlopt -verbose -g -ccopt -fPIC -c mod_wrap.c
    # linking of lib, including needed packages
    ocamlfind ocamlopt -ccopt -Wl,-Bsymbolic -verbose -o modlib.native.so -
ccopt -shared mod.cmx mod_wrap.o -package unix -package bigarray -linkpkg

4) using the shared library
    # compile a main:
    ocamlopt -verbose -g -ccopt -Iinclude -c main.c
    # link a main:
    gcc -g -o main main.o -l:modlib.native.so -L. -Wl,-rpath=.

However, dynamic linking has more troubles:

I used an approach simular to the one already used for libcamlrun_shared.so to 
end up with a libasmrun_shared.so.

Whenever I try to use this by means of using -output-obj and then linking with 
g++ I always end
up with a program that crashes in caml_oldify_local_roots
during GC in a non-trivial program.

I finally got it working correctly by running ocamlopt with the verbose flag and 
adapting
the output to do something like this:

gcc -o 'libembedded.so'   '-L/usr/lib/ocaml/threads' '-L/usr/lib/ocaml' 
'/tmp/camlstartup.o' '/usr/lib/ocaml/std_exit.o' 'client.o' 'client.a' 
'/usr/lib/ocaml/bigarray.a' '/usr/lib/ocaml/dbm.a' '/usr/lib/ocaml/str.a' 
'/usr/lib/ocaml/threads/threads.a' '/usr/lib/ocaml/unix.a' 
'/usr/lib/ocaml/stdlib.a' '-lbigarray' '-lmldbm' '-lgdbm_compat' '-lgdbm' '-
lstr' '-lthreadsnat' '-lpthread' '-lunix' 'client_wrap.o' -L/usr/lib/ocaml -
lasmrun_shared -lm  -ldl -L../../../target/foo/ -lfoo -L. -lbar -L. baz.o -
shared -Wl,-Bsymbolic -ggdb3 -Wall -Wextra
Where I first used on of the old camlstartup*.s that was still lying around and 
compiling that to camlstartup.o:
as -o /tmp/camlstartup.o /tmp/camlstartupaf7c1e.s

The end result seems to work without crashing. It seems to me that in order to 
have a usable libasmrun_shared.so some changes to the toolchain (ocamlopt) are 
needed.

Given this result I'll stick to statically linking in libasmrun.a until there 
is a proper solution within the toolchain. Unless someone steps in and tells 
me I did something wrong ;-)

Greetings and happy new year, Joost Damad

-- 
Joost Yervante Damad - http://damad.be/joost/