Dynamic link

Dynamic link

[Nicolas GEORGE]

I have written a small patch to the OCaml (2.99) run-time, that enables to
dynamically load C primitives, instead to have to link against a custom
run-time. It is far from perfect, but at this time, it works (under Unix
systems).

The main problem is that it's a bit hand made.

- the bytecode file has to be linked using the -use-prims undocumented
  option
- the run-time must be called with a -dfilename option, where filename is
  a primitive database (each line is the name of a .so file or of a symbol
  in the last .so file)
- instead of checking the constitency between the primitive list of the
  run-time and of the bytecode, it loads the needed symbols (here, I should
  have built a hash table instead of that stupid search)
- the libraries have to resolve themselves their external dependencies (I
  use ld -G -l*)
- there is a small memory waste (2kb) because the names of the builtins
  primitives are twice im memory

Of course, there are some points to solve. The linker should build himself
the list of needed primitives, but I was too affraid to touch the compiler.
I don't know how to make that work under windows (but I think it is
aproximativly the same) or mac, but it is not _necessary_.

One main advantage is to can use OCaml for small applications (such as
simple front-ends) that needs special libraries (GUI, Unix, Str...) without
the 1/2 megabyte of the custom run-time.

The patch is available at:
http://www.eleves.ens.fr:8080/home/george/informatique/ocaml-2.99-dl.patch

Here is an example of code:




J'ai écrit un petit patch pour le run-time OCaml (2.99), qui permet de
charder dynamiquement des primitives C, à la place d'avoir à lier contre un
run-time personnalisé. Il est loin d'être parfait, mais à l'heure actuelle,
il marche (sous des systèmes Unix).

Le principal problème est qu'il est un peu artisanal.

- le fichier de bytecode doit être lié en utilisant l'option
  non-docummentée -use-prims
- le run-time doit être appelé avec une option -dnomdefichier, où
  nomdefichier est une base de données de primitives (chaque ligne est le
  nom d'un fichier .so ou d'un symbole du dernier fichier .so)
- au lieu de vérifier la consistance entre les listes de primitives du
  run-time et du bytecode, il charge les symboles nécessaires (ici,
  j'aurais dû construire une table de hachage au lieu de cette recherche
  stupide)
- les bibliothèques doivent résoudre elles-mêmes leurs dépendances externes
  (J'utilise ld -G -l*)
- il y a un léger gaspillage de mémoire (2ko) parceque les noms des
  primitives internes sont deux fois en mémoire

Bien sûr, il y a des points à résoidre. L'éditeur de liens devrait
construire lui-même la liste des primitives nécessaires, mais j'avais trop
peur pour toucher au compilateur. Je ne sais pas comment faire marcher ça
sous windows (mais je crois que c'est approximativement la même chose) ou
sur mac, mais ce n'est pas _indispensable_.

Un principal avantage est de pouvoir utiliser OCaml pour de petites
applications (comme des front-ends simples) qui ont besoin de bibliothèques
spéciales (GUI, Unix, Str...) sans le 1/2 mégaoctet du run-time
personnalisé.

Le patch est disponible en :
http://www.eleves.ens.fr:8080/home/george/informatique/ocaml-2.99-dl.patch

Voici un exemple de code :



libtestpatch.c:
/*
gcc -I/usr/local/util/packages/ocaml-2.99/lib/ocaml \
`gtk-config --cflags` -O2 -c libtestpatch.c
ld -G -o libtestpatch.so libtestpatch.o `gtk-config --libs`
*/

#include <caml/mlvalues.h>
#include <caml/memory.h>
#include <gtk/gtk.h>

void ml_test(void)
{
  CAMLparam0();
  gtk_init(NULL,NULL);
  gtk_widget_show(gtk_window_new(GTK_WINDOW_TOPLEVEL));
  gtk_main();
  CAMLreturn(Val_unit);
}

testpatch.ml:
external test : unit -> unit = "ml_test"

let () = print_string "Bonjour";
  print_newline ();
  test ()

List of primitives to use with -use-prims:
ocamlrun -p > primlist.cmp
echo ml_test >> primlist.cmp

List of primitives to use with -d:
/some/where/libtestpatch.so
ml_test

Re: Dynamic link

[Jerome Vouillon]

On Fri, Apr 21, 2000 at 07:13:00PM +0200, Nicolas GEORGE wrote:
> I have written a small patch to the OCaml (2.99) run-time, that enables to
> dynamically load C primitives, instead to have to link against a custom
> run-time. It is far from perfect, but at this time, it works (under Unix
> systems).
[...]
> One main advantage is to can use OCaml for small applications (such as
> simple front-ends) that needs special libraries (GUI, Unix, Str...) without
> the 1/2 megabyte of the custom run-time.

Actually, if you are concerned about the size of a custom runtime, it
is much simpler to patch the OCaml makefiles so that the standard
library and the special libraries are built as shared library.

I think your patch would be much more interesting if it provided an
API for dynamically loading C libraries at execution time (rather than
at initialization time) and if the corresponding C primitives could
then be used by dynamically loaded Caml code.

[...]
> - there is a small memory waste (2kb) because the names of the builtins
>   primitives are twice im memory

I don't think you need to consider builtin primitives in a special
way: just consider the runtime as another shared library.

> Of course, there are some points to solve. The linker should build himself
> the list of needed primitives, but I was too afraid to touch the compiler.

You should definitively do that. And I think it's easy.

-- Jerome

Re: Dynamic link

[Nicolas GEORGE]

Le vendredi 21 avril 2000, Jerome Vouillon a écrit :
> Actually, if you are concerned about the size of a custom runtime, it
> is much simpler to patch the OCaml makefiles so that the standard
> library and the special libraries are built as shared library.

I agree with that, as long as you are root on your computer, to install the
new "fat" runtime each time you add a new library.


> API for dynamically loading C libraries at execution time (rather than

In my view, this is a quite different issue. (And I don't see immediatly
how to do that type safely)


> I don't think you need to consider builtin primitives in a special
> way: just consider the runtime as another shared library.

At this time, I am not at ease with the compiler/run-time code enough to
make so wide changes. But the structure of my patch allows that almost
without changes: I simply builtins primitives into the symbols database,
and as the symbol is already initialized, it does not load this fake
library.

> You should definitively do that. And I think it's easy.

I will try, but I have others parallel projects (I have written a binding
for the curses library, and I must then build on top of that a high-level
widget system).

-- 
  Sorry for my horrible English.

Re: Dynamic link

[Vitaly Lugovsky]

On Sat, 22 Apr 2000, Nicolas GEORGE wrote:

> > Actually, if you are concerned about the size of a custom runtime, it
> > is much simpler to patch the OCaml makefiles so that the standard
> > library and the special libraries are built as shared library.
> 
> I agree with that, as long as you are root on your computer, to install the
> new "fat" runtime each time you add a new library.

 What for? Why unprivileged user can't install'em in his own 
LD_LIBRARY_PATH? Or, if there is a group of users, e.g. "mlusers",
there can be a group writable directory for a shared libraries.

> > API for dynamically loading C libraries at execution time (rather than
> 
> In my view, this is a quite different issue. (And I don't see immediatly
> how to do that type safely)

 It'll be as safe as normal static link can be. Or I just can't understand
something...

--

   V.S.Lugovsky aka Mauhuur (http://ontil.ihep.su/~vsl) (UIN=45482254)

Re: Dynamic link

[Nicolas GEORGE]

Le mercredi 26 avril 2000, Vitaly Lugovsky a écrit :
> > I agree with that, as long as you are root on your computer, to install the
> > new "fat" runtime each time you add a new library.
>  What for? Why unprivileged user can't install'em in his own 
> LD_LIBRARY_PATH? Or, if there is a group of users, e.g. "mlusers",
> there can be a group writable directory for a shared libraries.

I've said root to be quick. What I meant was: "the same as the owner of the
compiler". Of course, every user can install his own version of the
run-time, with all the libraries, but that's exactly the problem. And
another problem is when you want to install a small application somewhere.
You won't rebuild a new run-time with the library, that's too heavy.

>  It'll be as safe as normal static link can be. Or I just can't understand
> something...

I wonder your method. Static link is checked by the compiler. My dynamic
link patch don't change anything as it does not bring any semantics more.
Explicit loading (that what is wanted, or did I bad understand?) of a C
primitive is something totally different, and rarely useful.

But as I re-read the mail, I see another interpretation. And here, I agree
that it would be useful. But I fear this would require a very large change
in the bytecode structure, because, as far as I have seen, the primitive is
stored as its index in the primitive table of the run-time.