Looking for pointers regarding integration of OCaml plugins into OCaml native code

Looking for pointers regarding integration of OCaml plugins into OCaml native code

[David MENTRE]

Hello,

I would like to extend an OCaml native code application with plugins
written in OCaml (preferably native code). So, much like the C/OCaml
interface, I would like to have some dynamically loaded OCaml code
calling my application core code and vice versa.

>From my readings of the caml-list archives, I understand that:

 1/ It is possible to load bytecode code into a bytecode application,
    using the Dynlink module;

 2/ It could be possible to load native code into a native code
    application[1] but Xavier thinks this is no longer possible or too
    difficult[2]. I haven't be able to find the explanation Xavier is
    refering to. Has anybody a pointer to it? I would like to understand
    the issue(s).

 3/ With original OCaml, it is not possible to load a bytecode into a
    native code application but that might be possible with the
    Asmdynlink module of Fabrice Le Fessant[2]. If I remember correctly
    (can't remember where I read that), the main issue is that native
    and bytecode have not exactly the same memory representation (thus,
    for example, the GC is different). Is that correct?

Beside the missing pointer in 2/, has anybody some pointers on the
issues involved?

Best whishes,
david

Footnotes:

[1] Feature 3 in:
http://groups.google.com/group/fa.caml/browse_frm/thread/2827ec9d553e7760/392f26ba4316e957?tvc=1&q=xavier+leroy+dynamic+load#392f26ba4316e957 

[2]  http://groups.google.com/group/fa.caml/browse_frm/thread/941afd4bad6fde92/865f61b89ec57b7a?q=xavier+leroy+dynamic+native+code&rnum=1#865f61b89ec57b7a

-- 
pub  1024D/A3AD7A2A 2004-10-03 David MENTRE <dmentre@linux-france.org>
 5996 CC46 4612 9CA4 3562  D7AC 6C67 9E96 A3AD 7A2A

Re: [Caml-list] Looking for pointers regarding integration of OCaml plugins into OCaml native code

[Basile STARYNKEVITCH]

Le Sat, Mar 04, 2006 at 02:38:32PM +0100, David MENTRE écrivait/wrote:
> Hello,
> 
> I would like to extend an OCaml native code application with plugins
> written in OCaml (preferably native code). So, much like the C/OCaml
> interface, I would like to have some dynamically loaded OCaml code
> calling my application core code and vice versa.
> 
> >>From my readings of the caml-list archives, I understand that:
> 
>  1/ It is possible to load bytecode code into a bytecode application,
>     using the Dynlink module;
> 
>  2/ It could be possible to load native code into a native code
>     application[1] but Xavier thinks this is no longer possible or too
>     difficult[2]. I haven't be able to find the explanation Xavier is
>     refering to. Has anybody a pointer to it? I would like to understand
>     the issue(s).

I'm not an expert on Ocaml's GC, but I believe the issue is the local
roots on the call stack. With bytecode, it is only on the VM stack,
whose organisation is well understood. With native code, the pointers
are on the machine's stack (the one usually used by C), and the Ocaml
generated code contains extra information to describe it (IIRC some
kind of hash tables mapping machine return addresses to stack frame
descriptions).

>  3/ With original OCaml, it is not possible to load a bytecode into a
>     native code application but that might be possible with the
>     Asmdynlink module of Fabrice Le Fessant[2]. If I remember correctly
>     (can't remember where I read that), the main issue is that native
>     and bytecode have not exactly the same memory representation (thus,
>     for example, the GC is different). Is that correct?

I believe it is related to the previous point.

There might be also another possibility. You might consider using
Metaocaml see http://www.metaocaml.org/ with the following caveats

  First and above all, Metaocaml is much less mature than Ocaml is,
  and the team of people working on Metaocaml is much smaller. This
  means that Metaocaml is really a research prototype, not a full
  software product (there is no offense intended in stating this fact).

  MetaOcaml is native only on x86 (32 bits). IIRC, it is not native on
  x86_64 (aka AMD64) nor on all the other native targets of Ocaml.

  IIRC, there is not robust machinery to drop useless code, which is
needed for long-time running programs in MetaOcaml. Basically, what is
missing (both in MetaOcaml & in Ocaml) is the ability to garbage
collect useless code. But this requirement has a major impact on the
compiler, the runtime library and the performance of the GC,
etc... Maybe Ocaml's successor would have the meta-programming
abilities of MetaOcaml and a way to drop useless code, ie a GC
handling machine code (old versions of SML/NJ did have it, and some
CommonLisp implementations have it also). There are also some
theoretical barriers (see all the work on safe module signatures,
mixins, mobility, ....).

  I'm not sure that MetaOcaml has the ability to do exactly a module
run-time linking; however, it is able to generate code at runtime
which might be somehow equivalent. It could recieve some description
of the code to generate...

Maybe the easiest way is to stay within the bytecode... Do you really
need the performance of native code? You might also try with
ocamljitrun (but I admit that I don't have much time maintaining it).

Regards.
-- 
Basile STARYNKEVITCH         http://starynkevitch.net/Basile/ 
email: basile<at>starynkevitch<dot>net 
aliases: basile<at>tunes<dot>org = bstarynk<at>nerim<dot>net
8, rue de la Faïencerie, 92340 Bourg La Reine, France

Re: [Caml-list] Looking for pointers regarding integration of OCaml plugins into OCaml native code

[Alessandro Baretta]

David MENTRE wrote:
> Hello,
>
>  2/ It could be possible to load native code into a native code
>     application[1] but Xavier thinks this is no longer possible or too
>     difficult[2]. I haven't be able to find the explanation Xavier is
>     refering to. Has anybody a pointer to it? I would like to understand
>     the issue(s).

Actually, MetaOcaml has most of the machinery needed to do this. I have recently 
released to the MetaOcaml hackers list a patch which actually enables native 
linking of ocamlopt generated code on Linux/x86. I have no clue as to the level 
of support for any other operating system/architecture. I must add that, 
although Natdynlink exists in MetaOcaml, it is definitely not a mature 
implementation. I had to solve several issues myself before I could get my 
AS/Xcaml to run any bit of a web application in native dynamic mode. Yet, after 
all the bugfixing, I only have a proof-of-concept implementation, which is not 
stable enough for production use. A lot more testing and feedback is needed the 
by the MetaOcamlers to get this feature to work right.

And, by the way, MetaOcaml is really much more stable than its official "alpha" 
status implies. As far as I seen while experimenting with building the AS/Xcaml, 
is that Natdynlink needs more work. The rest seems fairly robust.

Alex



-- 
*********************************************************************

Ing. Alessandro Baretta

Studio Baretta
http://studio.baretta.com/

Consulenza Tecnologica e Ingegneria Industriale
Technological Consulting and Industrial Engineering

tel. +39 02 370 111 55
fax. +39 02 370 111 54

Re: [Caml-list] Looking for pointers regarding integration of OCaml plugins into OCaml native code

[Walid Taha]

This would be a very interesting approach to explore.  Basile presented
pretty convincing numbers on ocamljit a couple of years ago.  Another
advantage of this approach is that if you also want to compile at run time
(with MetaOCaml), the bytecode compiler is much faster to compile than the
native code one.

Walid.

On Sat, 4 Mar 2006, Basile STARYNKEVITCH wrote:

|Maybe the easiest way is to stay within the bytecode... Do you really
|need the performance of native code? You might also try with
|ocamljitrun (but I admit that I don't have much time maintaining it).

Re: [Caml-list] Looking for pointers regarding integration of OCaml plugins into OCaml native code

[David MENTRE]

Hello Basile,

Basile STARYNKEVITCH <basile@starynkevitch.net> writes:

> Maybe the easiest way is to stay within the bytecode... Do you really
> need the performance of native code? You might also try with
> ocamljitrun (but I admit that I don't have much time maintaining it).

Yes, ocamljitrun offers in interesting fourth alternative.

In fact, I don't really need right now this facility. But I'm a bit
frustrated, when writing a program in OCaml using native code compiler,
to be unable to extend it with dynamically loadable OCaml code. It is
easier to write plugins for an OCaml program in Python, Perl or Lua than
in OCaml. :-)

Best wishes,
d.
-- 
pub  1024D/A3AD7A2A 2004-10-03 David MENTRE <dmentre@linux-france.org>
 5996 CC46 4612 9CA4 3562  D7AC 6C67 9E96 A3AD 7A2A

Re: [Caml-list] Looking for pointers regarding integration of OCaml plugins into OCaml native code

[Nathaniel Gray]

On 3/11/06, David MENTRE <dmentre@linux-france.org> wrote:
>
> In fact, I don't really need right now this facility. But I'm a bit
> frustrated, when writing a program in OCaml using native code compiler,
> to be unable to extend it with dynamically loadable OCaml code. It is
> easier to write plugins for an OCaml program in Python, Perl or Lua than
> in OCaml. :-)

I agree.  I would say this is my #1 apprehension when I consider
writing any sort of consumer-level app (e.g. browser, mail client,
etc) in OCaml.  You might check out Nemerle if this is important to
you.  It's similar to OCaml in many ways but compiles to .NET
bytecode.

Cheers,
-n8

--
>>>-- Nathaniel Gray -- Caltech Computer Science ------>
>>>-- Mojave Project -- http://mojave.cs.caltech.edu -->