Dynamic loading of native code : what about librairies and packs ?

Dynamic loading of native code : what about librairies and packs ?

[Pierre-Loïc Garoche]

[-- Attachment #1: Type: text/plain, Size: 1463 bytes --]

I am discovering this feature of loading dynamically native code. The
Frama-C framework (www.frama-c.cea.fr) provides now a plugin
architecture that allows you to develop your own plugin and loads it
directly in the framework.

I targeted to adapt a simple tool I developped as a Frama-C plugin and
faced some difficulties. I did not found enough information about the
compilation of cmxs files.

** First: what about external libraries ?
Could you detail the behavior of the linking process of cmxs file with
respect to dependancies.

If I understand well any library used to compile cmx files should be
- either know by the software dynamically loading the plugin,
- or its cmxa should be passed as argument of ocamlopt -shared command.

But what is the difference between:
   ocamlopt.opt -shared  mylib.cmxa xxx.cmx yyy.cmx -o target.cmxs
and
   ocamlopt.opt -shared -linkall mylib.cmxa xxx.cmx yyy.cmx -o target.cmxs

Both seems to work similarly on my example (no differences between
resulting files).


** Second: how about packs ?
Does the option "-shared -o target.cmxs" is similar to "-pack -o
target.cmx", without the dynamic loading thing ? And what if the plugin
components (cmx files) are or are not compiled with the "-for-pack" option ?


Any information would be greatly appreciated.

Best regards,

pl
-- 
Pierre-Loïc Garoche
pierre-loic.garoche@onera.fr
http://www.onera.fr/staff/pierre-loic-garoche/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Alain Frisch]

Pierre-Loïc Garoche wrote:
> ** First: what about external libraries ?
> Could you detail the behavior of the linking process of cmxs file with
> respect to dependancies.
> 
> If I understand well any library used to compile cmx files should be
> - either know by the software dynamically loading the plugin,
> - or its cmxa should be passed as argument of ocamlopt -shared command.
> 
> But what is the difference between:
>    ocamlopt.opt -shared  mylib.cmxa xxx.cmx yyy.cmx -o target.cmxs
> and
>    ocamlopt.opt -shared -linkall mylib.cmxa xxx.cmx yyy.cmx -o target.cmxs
> 
> Both seems to work similarly on my example (no differences between
> resulting files).

The -linkall forces all the modules in the included libraries (here, 
mylib.cmxa) to be included in the .cmxs file, even if they are not 
explicitly used by other linked modules. This option is generally 
useless when linking cmxs files, unless the .cmxs file is loaded in 
non-private mode, and you want subsequently loaded .cmxs files to be 
able to access any module in the libraries. (Typically, if you turn 
mylib.cmxa into a mylib.cmxs, without any other module, you want -linkall.)


> ** Second: how about packs ?

As far as I can tell, packing and creating cmxs files are orthogonal 
features: they don't overlap and they don't have bad interactions. It is 
ok to put in cmxs files a module produced by -pack. I think it is also 
ok to put modules compiled with -for-pack, but I don't see why you would 
do that.



Regards,


Alain

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Pierre-Loïc Garoche]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

>> ** Second: how about packs ?
> 
> As far as I can tell, packing and creating cmxs files are orthogonal
> features: they don't overlap and they don't have bad interactions. It is
> ok to put in cmxs files a module produced by -pack. I think it is also
> ok to put modules compiled with -for-pack, but I don't see why you would
> do that.

According to the manual,
"The options -pack, -a, -shared, -c and -output-obj are mutually exclusive."

Do you suggest to create a first cmx pack and then make it dynamically
loadable ?

Regards,

pl
-- 
Pierre-Loïc Garoche
pierre-loic.garoche@onera.fr
http://www.onera.fr/staff/pierre-loic-garoche/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Alain Frisch]

Pierre-Loïc Garoche wrote:
> Do you suggest to create a first cmx pack and then make it dynamically
> loadable ?

Yes, absolutely.

-- Alain

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Pierre-Loïc Garoche]

[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]

Hi Alain, Julien and all other Ocaml list members,

I am still having some problems with the dynamic loading of native code. Allow me to give you an extremely simple example to illustrate my problem. I hope you can clarify my understanding of it.

My problem concerns the dynamic loading of native code where the dynamic code loaded depends on another library.

Basically there are three files: 
- main.ml, dynamically loading the plugin
- plugin.ml, the loaded code that depends on the external lib
- mylib.ml, the external lib

///////
main.ml:
let _ = print_string "main\n"
let _ = Dynlink.loadfile "MyPlugin.cmxs"

compiled with ocamlopt -o MyProg dynlink.cmxa main.ml

///////
mylib.ml:
let _ = print_string "mylib\n"
let myval : (int , int) Hashtbl.t = Hashtbl.create 13

compiled with ocamlopt -a -linkall -o mylib.cmxa mylib.ml

//////
plugin.ml:
let _ = print_string "plugin\n"
let cst = Mylib.myval

compiled and linked to build a shared library with
ocamlopt -shared -linkall -o MyPlugin.cmxs mylib.cmxa plugin.ml

Running it gives me the following error
error loading shared library: blabla/MyPlugin.cmxs: undefined symbol: camlHashtbl__create_79


Remark1 : Of course, building a standalone plugin works:
ocamlopt -o PluginSelf mylib.cmxa plugin.cmx

Remark2: If I don't rely on an external module and replace myval by an integer or any other self defined type value, it works as well.

Remark3: The linkall option does not seems to have an impact on such simple example.


Question: How should I link it to rely on external libraries and produce a valid MyPlugin.cmxs ?

Any hint or comment will be greatly appreciated !

Best regards,

pl


-- 
Pierre-Loïc Garoche
pierre-loic.garoche@onera.fr
http://www.onera.fr/staff/pierre-loic-garoche/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Christophe TROESTLER]

On Tue, 1 Sep 2009 10:56:47 +0200, Pierre-Loïc Garoche wrote:
> 
> Question: How should I link it to rely on external libraries and
> produce a valid MyPlugin.cmxs ?

You should reference « Hashtbl » in your main program :

> main.ml:
  module ForLinking_1 = Hashtbl
  let () = print_string "main\n"
  let () = Dynlink.loadfile "MyPlugin.cmxs"

BTW, if you want your program to work in both bytecode and native
code, you should use:

  Dynlink.loadfile (Dynlink.adapt_filename "MyPlugin.cmo")

Cheers,
C.

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Alain Frisch]

Pierre-Loïc Garoche wrote:
> error loading shared library: blabla/MyPlugin.cmxs: undefined symbol: camlHashtbl__create_79

You need to ensure that the main program is linked with all the modules 
that are needed by the dynamically loaded modules (including the modules 
from the standard library). Linking the main program with -linkall 
should solve your problem.

-- Alain

Re: [Caml-list] Dynamic loading of native code : what about librairies and packs ?

[Julien Signoles]

Hello,

> I am discovering this feature of loading dynamically native code. The
> Frama-C framework (www.frama-c.cea.fr) provides now a plugin
> architecture that allows you to develop your own plugin and loads it
> directly in the framework.
> 
> I targeted to adapt a simple tool I developped as a Frama-C plugin and
> faced some difficulties. I did not found enough information about the
> compilation of cmxs files.

Alain's answers are very good. I confirm that packing and dynamic
loading are compatible features: all Frama-C plug-ins are packed and
some of them are dynamically loaded.

However, in the current distributed release of Frama-C, dynamic loading
of plug-ins is still experimental. This feature will be improved in the
next release and it should be easier for plug-in developers to compile
dynamic plug-ins.

Anyway, if you have any additional Frama-C-specific questions/feedbacks
about plug-in development, don't hesitate to sent a message on the
Frama-C public mailing list
(http://lists.gforge.inria.fr/cgi-bin/mailman/listinfo/frama-c-discuss).

Br,
Julien Signoles
-- 
Researcher-engineer
CEA LIST, Software Reliability Lab
91191 Gif-Sur-Yvette Cedex
tel:(+33)1.69.08.71.83  fax:(+33)1.69.08.83.95  Julien.Signoles@cea.fr