Using dynlink (native code and byte code) for the build system

Using dynlink (native code and byte code) for the build system

[Berke Durak]

Hello all,

Thank to Alain Frisch's natdynlink, it is now possible to dynamically load native code plugins.
We were thinking with Nicolas of dynamically of using this to loading all plugins dynamically, instead of linking them
with ocamlbuild and re-running the resulting executable as is done currently.  This would allow to load any number of 
plugins during the course of the compilation.

It has been recently suggested on this list that instead of forcing everyone to use the same build system, we could ask 
them to have a common build API.

One proposition was a makefile with a fixed number of targets (byte, native, clean...)

I think that we should minimize the set of different languages used per the ``you shall not unnecessarily multiply 
entities'' principle.  Also, Makefiles are not very portable, because:

   * There are different implementations of make (BSD make vs GNU make)
   * Makefiles embed shell commands and filenames, which cause huge portability problems

So why not require projects to have a buld API in an Ocaml file?  The API would be distinct from ocamlbuild but would
be designed to be extensible, to permit dynamic loading and to be callable from ocamlbuild.

I have started a toy system.  Every project includes in its root a file named plugin.ml, which contanis something such as:

open Interface

class entry env =
   object
     inherit default_builder

     method hello =
       env.env_say "Hello, world!"

     method build =
       env.env_say "Building";
       Sys.command "make linux"

     method clean =
       env.env_say "Cleaning";
       Sys.command "make clean"
   end

let _ = register (fun env -> new entry env)

Interface is the API we are talking of.  The plugin registers an object of class entry.
The entry object has a #build method which is called to build and install the project,
according to options and target directories defined in the environment env.

The environment contains a registry-like configuration dictionary whose interpretation
is project-dependent.

The entry object can also have a #prerequisites method which returns a description of
the dependencies of the project.

I've made a toy API and a toy builder which compiles and loads native plugins,
and written a plugin for a C project which is built using make.  You can browse
the source at

   http://aurochs.fr/hg/opkg/

or clone it using Mercurial with the command:

   hg clone http://aurochs.fr/hg/opkg/

You then run ./make and then ./builder

So this is a request for experimenting and developing a plugin system around these ides.
Feel absolutely free to do anything based on the opkg sample code.  I hope a nice plugin system will
evolve from it.

PS. You can commit in your own repository;  if you want to have push access
just send me a mail with the result of htpasswd -nd yourlogin

PPS. I put the code in the public domain.
-- 
Berke DURAK

Re: [Caml-list] Using dynlink (native code and byte code) for the build system

[Jon Harrop]

On Thursday 31 January 2008 14:49:44 Alain Frisch wrote:
> Berke Durak wrote:
> > Thank to Alain Frisch's natdynlink, it is now possible to dynamically
> > load native code plugins.
>
> Only on some platforms! Please don't make the availability of exotic
> features mandatory for the build system to work.

FWIW, I think natdynlink should be a core feature of OCaml going forwards 
because it is useful in so many circumstances, not least the performant 
top-level.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Using dynlink (native code and byte code) for the build system

[Alain Frisch]

Berke Durak wrote:
> Thank to Alain Frisch's natdynlink, it is now possible to dynamically 
> load native code plugins.

Only on some platforms! Please don't make the availability of exotic 
features mandatory for the build system to work.

-- Alain

Re: [Caml-list] Using dynlink (native code and byte code) for the build system

[Jon Harrop]

On Thursday 31 January 2008 14:59:28 Berke Durak wrote:
> Jon Harrop a écrit :
> > On Thursday 31 January 2008 14:49:44 Alain Frisch wrote:
> >> Berke Durak wrote:
> >>> Thank to Alain Frisch's natdynlink, it is now possible to dynamically
> >>> load native code plugins.
> >>
> >> Only on some platforms! Please don't make the availability of exotic
> >> features mandatory for the build system to work.
> >
> > FWIW, I think natdynlink should be a core feature of OCaml going forwards
> > because it is useful in so many circumstances
>
> I agree.  However such a build system will also work with bytecode, albeit
> at lower performance, on platforms not supporting natdynlink.  A build
> system is also not computationally too intensive, so it should be OK.

Good point. AMD64 gives far better performance than all other platforms anyway 
(AFAIK) so people should be using that if they want good performance anyway.

>  > not least the performant top-level.
>
> Natdynlink in the toplevel??

Alain wrote "ocamlnat", an OCaml top-level that uses compilation to native 
code (with dynamic linking) to provide performance competitive with other 
languages like Lisp and F#. I found this extremely useful for the few months 
that I used it, although it was a little buggy.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] Using dynlink (native code and byte code) for the build system

[Berke Durak]

Jon Harrop a écrit :
> On Thursday 31 January 2008 14:49:44 Alain Frisch wrote:
>> Berke Durak wrote:
>>> Thank to Alain Frisch's natdynlink, it is now possible to dynamically
>>> load native code plugins.
>> Only on some platforms! Please don't make the availability of exotic
>> features mandatory for the build system to work.
> 
> FWIW, I think natdynlink should be a core feature of OCaml going forwards 
> because it is useful in so many circumstances
> 

I agree.  However such a build system will also work with bytecode, albeit at lower performance,
on platforms not supporting natdynlink.  A build system is also not computationally too intensive,
so it should be OK.

 > not least the performant top-level.

Natdynlink in the toplevel??
-- 
Berke DURAK

Re: [Caml-list] Using dynlink (native code and byte code) for the build system

[Alain Frisch]

Jon Harrop wrote:
> FWIW, I think natdynlink should be a core feature of OCaml going forwards 
> because it is useful in so many circumstances, not least the performant 
> top-level.

As you know, the current implementation of natdynlink relies on the 
dynamic library layer of the underlying OS. Some systems don't have 
dynamic libraries at all and some systems put strong constraints on what 
can be put in them (e.g. "no reference to undefined symbols" or 
"position-independent code"), which require changes to the ocamlopt 
backend or other hacks.

A nicer version of natdynlink (and of the native toplevel) could be 
obtained by basing it on an LLVM backend for ocamlopt. (And I'm sure the 
MetaOCaml guys would also love such a backend.)

-- Alain

Re: [Caml-list] Using dynlink (native code and byte code) for the build system

[Jon Harrop]

On Thursday 31 January 2008 16:46:37 Alain Frisch wrote:
> A nicer version of natdynlink (and of the native toplevel) could be
> obtained by basing it on an LLVM backend for ocamlopt. (And I'm sure the
> MetaOCaml guys would also love such a backend.)

Absolutely!

A reusable LLVM backend for ocamlopt will be a major step forward in OCaml's 
evolution, greatly broadening OCaml's usefulness as a tool.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e