Native dynlink on 3.11: a request for packagers

Native dynlink on 3.11: a request for packagers

[Dario Teixeira]

Hi,

As you are probably aware, the upcoming 3.11 release includes
Alain Frisch's native dynlink patches.  AFAIK, you cannot
directly dynlink a cmxa or cmx file like you would do in
bytecode with cma or cmo files.  Beforehand, a native code
binary must be made into a plugin via ocamlopt's -shared
option.  The suggested convention [1] is to give these
plugins a cmxs extension:

ocamlopt -shared -linkall -o foobar.cmxs foobar.cmxa

These native code plugins are a prerequisite if one intends
to run Ocsigen in native mode.  Therefore, the life of Ocsigen
users would be greatly simplified if library writers and
packagers (GODI, Debian, Fedora, etc) would add this little
extra step of generating cmxs for each cmxa.  This request
only makes sense for 3.11, of course.

Complementary, META files should also include extra directives
for plugins.  For example:

archive(byte) = "foobar.cma"
archive(native) = "foobar.cmxa"
archive(plugin,byte) = "foobar.cma"
archive(plugin,native) = "foobar.cmxs"

This request can of course become an OSR.  My question is if the
herein contained instructions will still be valid verbatim for
3.11 final.  Will they?  Also, please check the Ocsigen Wiki for
more info on this subject [2].

Best regards,
Dario Teixeira

[1] http://alain.frisch.fr/natdynlink.html
[2] http://www.ocsigen.org/trac/wiki/NativeCodeVersion

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Richard Jones]

Does ocamlfind support any of this?  Since I'm now also cross-
compiling OCaml, I've certainly come to appreciate findlib more than
ever.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Dario Teixeira]

Hi,

> Does ocamlfind support any of this?  Since I'm now also
> cross-compiling OCaml, I've certainly come to appreciate
> findlib more than ever.

Indeed it does.  The current version of Ocsigen already
makes use of this feature.  However, because very few
(none, actually, other than Ocsigen's) packages currently
ship with cmxs files, an Ocsigen user who wants to play
with native code is forced to manually generate them.
(What impelled me to write the request was precisely
having just gone through the process of producing the
cmxs for a bunch of Ocamlnet and PXP libraries...)

Cheers,
Dario

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Jon Harrop]

On Friday 21 November 2008 23:45:34 Dario Teixeira wrote:
> Hi,
>
> > Does ocamlfind support any of this?  Since I'm now also
> > cross-compiling OCaml, I've certainly come to appreciate
> > findlib more than ever.
>
> Indeed it does.  The current version of Ocsigen already
> makes use of this feature.  However, because very few
> (none, actually, other than Ocsigen's) packages currently
> ship with cmxs files, an Ocsigen user who wants to play
> with native code is forced to manually generate them.
> (What impelled me to write the request was precisely
> having just gone through the process of producing the
> cmxs for a bunch of Ocamlnet and PXP libraries...)

This begs the question: what is the advantage of a .cmxa over a .cmxs, and 
what are the downsides of linking everything dynamically?

-- 
Dr Jon Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/?e

RE: [Caml-list] Native dynlink on 3.11: a request for packagers

[David Allsopp]

On 22 November 2008 01:20 Jon Harrop wrote:
> On Friday 21 November 2008 23:45:34 Dario Teixeira wrote:
> > Hi,
> >
> > > Does ocamlfind support any of this?  Since I'm now also
> > > cross-compiling OCaml, I've certainly come to appreciate
> > > findlib more than ever.
> >
> > Indeed it does.  The current version of Ocsigen already
> > makes use of this feature.  However, because very few
> > (none, actually, other than Ocsigen's) packages currently
> > ship with cmxs files, an Ocsigen user who wants to play
> > with native code is forced to manually generate them.
> > (What impelled me to write the request was precisely
> > having just gone through the process of producing the
> > cmxs for a bunch of Ocamlnet and PXP libraries...)
> 
> This begs the question: what is the advantage of a .cmxa over a .cmxs,
> and
> what are the downsides of linking everything dynamically?

That's reducible to the standard debate of static vs dynamic linking in any language, surely? The only "impact" on performance presumably is the re-writing of the symbol tables which flexlink/dlopen does as the .cmxs file is loaded but that must be negligible. However, isn't ocamlopt able to do some cross-module optimisation (which is why .cmx files are usually shipped as well as .cmxa) - this presumably isn't possible when linking dynamically...

Short-term, native dynlink doesn't work on all platforms (AFAIK?) so .cmxa is not dead, whatever your linking preference.


David

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Stefano Zacchiroli]

On Fri, Nov 21, 2008 at 11:29:12AM -0800, Dario Teixeira wrote:
> Therefore, the life of Ocsigen users would be greatly simplified if
> library writers and packagers (GODI, Debian, Fedora, etc) would add
> this little extra step of generating cmxs for each cmxa.  This
> request only makes sense for 3.11, of course.

Debian-side, this is already planned. The work-flow will be as usual:
patch the Makefiles of upstream not (yet) using dynlinking, and send
patches upstream. Of course it will take some time, and library
authors helping out making new releases supporting that out of the box
will be really appreciated.

Cheers.

-- 
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Daniel Bünzli]

I think I'm missing something here. From the doc of Dynlink :

> No facilities are provided to access value names defined by the  
> unit. Therefore, the unit must register itself its entry points with  
> the main program, e.g. by modifying tables of functions.

Thus just generating a cmxs for a module won't allow you to use it  
dynamically. You'll need support from the module no ?

Daniel

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Gerd Stolpmann]

Am Freitag, den 21.11.2008, 15:45 -0800 schrieb Dario Teixeira:
> Hi,
> 
> > Does ocamlfind support any of this?  Since I'm now also
> > cross-compiling OCaml, I've certainly come to appreciate
> > findlib more than ever.
> 
> Indeed it does.  The current version of Ocsigen already
> makes use of this feature.

Ahem. I'm not aware of this feature... Can you please elaborate to what
you refer?

Gerd


>   However, because very few
> (none, actually, other than Ocsigen's) packages currently
> ship with cmxs files, an Ocsigen user who wants to play
> with native code is forced to manually generate them.
> (What impelled me to write the request was precisely
> having just gone through the process of producing the
> cmxs for a bunch of Ocamlnet and PXP libraries...)
> 
> Cheers,
> Dario
> 
> 
> 
>       
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Alain Frisch]

On 11/22/2008 4:00 PM, Daniel Bünzli wrote:
> Thus just generating a cmxs for a module won't allow you to use it
> dynamically. You'll need support from the module no ?

Well, if your program is only composed of .cmxs plugins (libraries + 
your own program), then you can have a generic driver that simply load a 
list of .cmxs files specified e.g. on its command line.

The point is that a plugin can refer to symbols defined in other plugins 
already dynlinked.

-- Alain

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Dario Teixeira]

Hi,

> > Indeed it does.  The current version of Ocsigen
> > already makes use of this feature.
> 
> Ahem. I'm not aware of this feature... Can you please
> elaborate to what you refer?

AFAIK, you can annotate findlib's META files with custom
tags.  Ocsigen expects dynamically loaded plugins to have
a "plugin" tag, like this:

archive(native) = "foobar.cmxa"
archive(plugin,native) = "foobar.cmxs"

Ocsigen's module loader can thus leverage all of Findlib's
existing facilities (such as dependency handling) also for
dynamically loading plugins.

(The Ocsigen guys can correct/expand on this better than
anyone, of course).

Cheers,
Dario

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Daniel Bünzli]

Le 21 nov. 08 à 20:29, Dario Teixeira a écrit :

> AFAIK, you cannot directly dynlink a cmxa or cmx file like you would  
> do in
> bytecode with cma or cmo files.  Beforehand, a native code
> binary must be made into a plugin via ocamlopt's -shared
> option.

To make it easy to do this for a module or a library would it be  
possible to add a new default %.cmxs target in ocamlbuild before 3.11  
is released (or is it already too late) ?

Best,

Daniel

P.S. Tell me if you prefer to have that in the bug tracker.

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Dario Teixeira]

Hi,

> To make it easy to do this for a module or a library would
> it be possible to add a new default %.cmxs target in
> ocamlbuild before 3.11 is released (or is it already too
> late) ?

Good point.  It shouldn't be too late, since the changes are
minimal and 3.11 is only at RC1 status...

Cheers,
Dario

Re: [Caml-list] Native dynlink on 3.11: a request for packagers

[Stefano Zacchiroli]

On Tue, Nov 25, 2008 at 08:36:28PM +0100, Daniel Bünzli wrote:
> To make it easy to do this for a module or a library would it be
> possible to add a new default %.cmxs target in ocamlbuild before
> 3.11 is released (or is it already too late) ?

Good idea. ... even though I doubt in practice it would impact
significantly on the speed of %.cmxs spreading, as the projects using
ocamlbuild to build are not that many (yet).

> P.S. Tell me if you prefer to have that in the bug tracker.

It is probably better, even only to increase the visibility and
traceability of the request.

Cheers.

-- 
Stefano Zacchiroli -o- PhD in Computer Science \ PostDoc @ Univ. Paris 7
zack@{upsilon.cc,pps.jussieu.fr,debian.org} -<>- http://upsilon.cc/zack/
Dietro un grande uomo c'è ..|  .  |. Et ne m'en veux pas si je te tutoie
sempre uno zaino ...........| ..: |.... Je dis tu à tous ceux que j'aime