[Caml-list] any automated FFI bindings generators?

[Caml-list] any automated FFI bindings generators?

[Ömer Sinan Ağacan]

Hi all,

I want to be able to use some very big C libraries from OCaml and I
want to automate process of writing bindings as much as possible. What
are my options for this? Do we have any tools to generate bindings?

Thanks,
Ömer.

Re: [Caml-list] any automated FFI bindings generators?

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 1677 bytes --]

Am Donnerstag, den 30.01.2014, 13:01 +0200 schrieb Ömer Sinan Ağacan:
> Hi all,
> 
> I want to be able to use some very big C libraries from OCaml and I
> want to automate process of writing bindings as much as possible. What
> are my options for this? Do we have any tools to generate bindings?

I ran into this question a couple of months ago. The options we have are
not really good (e.g. I tried camlidl before). Because of this I wrote
my own utility that is special-cased for my application (bindings of
gnutls):

https://godirepo.camlcity.org/wwwsvn/branches/onet4/code/tools/stubgen.ml?rev=1935&root=lib-ocamlnet2&view=auto

See here for an example how to use it:

https://godirepo.camlcity.org/wwwsvn/branches/onet4/code/src/nettls-gnutls/?root=lib-ocamlnet2

especially gnutls.descr. Basically, this a little OCaml program, and by
calling a function (called "standard") you generate a binding, e.g. 

standard "gnutls_alert_description_t gnutls_alert_get (gnutls_session_t
session)"

The syntax in the string mimicks C header declarations, and can use
special data types like ztstr (for zero-terminated string).

I haven't found time to make something more systematic out of this idea.
Currently, the strong point of this method is that it is quite easy to
adapt it to your own needs.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: [Caml-list] any automated FFI bindings generators?

[Francois Berenger]

On 01/30/2014 08:01 PM, Ömer Sinan Ağacan wrote:
> Hi all,
>
> I want to be able to use some very big C libraries from OCaml and I
> want to automate process of writing bindings as much as possible. What
> are my options for this? Do we have any tools to generate bindings?

I am not sure this is what you are looking for, but there is this in OPAM:

ctypes -- Combinators for binding to C libraries without writing any C

https://github.com/ocamllabs/ocaml-ctypes

There is also this:

cstruct -- access C structures via a camlp4 extension

-- 
Best regards,
Francois Berenger.

Re: [Caml-list] any automated FFI bindings generators?

[Adrien Nader]

Hi,

On Thu, Jan 30, 2014, Ömer Sinan Ağacan wrote:
> Hi all,
> 
> I want to be able to use some very big C libraries from OCaml and I
> want to automate process of writing bindings as much as possible. What
> are my options for this? Do we have any tools to generate bindings?

Some time ago I've started a project named "cowboy" (because it makes
bindings...) which could match what you're looking for. I've been using
it to generate lablgtk-compatible bindings to webkit-gtk and more
generally glib-based libraries since they all share the same
conventions.

In a few words, it's yacfe-light, an AST-simplification layer and a
custom output.

Now, for the full explanation.

1- Yacfe-light

Yacfe-light is a parser for unpreprocessed C (and C++ and Java). This
means it can extract more programmer-level information than other
parsers. Consider these code excerpts from my
/usr/include/webkit/webkitwebview.h:

  #include <webkit/webkitwebbackforwardlist.h>
  #include <webkit/webkitwebframe.h>
  #include <webkit/webkitwebhistoryitem.h>

This module depends on "webkitwebbackforwardlist", "webkitwebframe",
"webkitwebhistoryitem" (and a few others).

  #define WEBKIT_TYPE_WEB_VIEW (webkit_web_view_get_type())

We can also see that the short name of the module is "WEB_VIEW".

  struct _WebKitWebViewClass {
      GtkContainerClass parent_class;

      /*< public >*/
      [...]

      /* internal */
      void                       (* set_scroll_adjustments) (WebKitWebView        *web_view,
      [...]

      /* Padding for future expansion */
      void (*_webkit_reserved0) (void);
      [...]
  }

struct _WebKitWebViewClass is the central object to this library. It has
a "parent_class" which is used for inheritance in glib-based libraries.
Some of the elements in that class are for internal use and some of
them are merely padding for future extension of the object without
changing its ABI.

  #if !defined(WEBKIT_DISABLE_DEPRECATED)
  WEBKIT_API GdkPixbuf *
  webkit_web_view_get_icon_pixbuf (WebKitWebView *web_view);
  #endif

And one last bit is there is an API (as seen by the "WEBKIT_API"
attribute) which is available but deprecated.

Most of these bits would go away with a regular C parser which requires
the code to be preprocessed through 'cpp' first: there would be many
more lines of code, names and comments wouldn't be preserved, macros
which mean something to the programmer would be expanded to an
unreadable form and some lines would be dropped.

2- AST simplication layer
Yacfe-light is great. But it's a parser for a quite large language and
it tries to extract as much information as possible. It also cares about
the implementation of functions while this doesn't matter for binding
generation.

That's why cowboy has a layer to strip most of it and offer something
simpler.

3- Output
The last step is to output the actual bindings. I believe that large and
mature libraries require a specific backend. They have their own
API-style and it shouldn't be handled in a generic way since it would
make the bindings much lower-level and less pleasant to use.

In practice, for my glib backend which outputs code to be used with
lablgtk, this means (output of 'wc -l *.ml'):
  128 glib.ml -> main module which call others
   29 glibAnnots.ml -> don't remember
   71 glibC.ml -> .c file with the low-level code
   98 glibFixes.ml -> work-arounds for inconsistencies in the C libs
  158 glibG.ml -> g${Library}.ml file (lablgtk convention)
   68 glibGtk.ml -> gtk${Library}.ml file (lablgtk convention)
   23 glibGtkTypes.ml -> outputs type declarations
   27 glibH.ml -> .h file with type conversion macros (Val_foo())
  190 glibLasso.ml -> parsing of names following the glib conventions
   14 glibMETA.ml -> outputs a META file for use with ocamlfind
   32 glibOCaml.ml -> translate between C and ocaml type names
   45 glibOasis.ml -> outputs a _oasis file
   27 glibObjects.ml -> don't remember
  240 glibProps.ml -> outputs a ".props" file which is parsed by a
                      lablgtk tool which then outputs several files
   83 glibVar.ml -> outputs a ".var" file which is parsed by a
                    lablgtk tool which then outputs several files

Glib and lablgtk compat are probably more than what most large libraries
would require though and you can get something useful in way fewer lines
than that. The most annoying bit was definitely the names in glib-based
libraries: alternating between "WEBKIT_WEB_VIEW", "WebKitWebView",
"webkit_web_view_*", ... telling the code to understand that as "web"
and "kit" or as "webkit".

I also haven't had much trouble with updating and newer versions of the
C libraries and have been fairly happy to have spent some time working
on the automation.


Caveats:
I haven't been able to work on it recently and the code could most
probably be improved but it the codebase isn't huge either and it
doesn't have dead corpses in it.

Yacfe-light doesn't enjoy C++ that much. This means that a .cpp file
will make it choke even though the only functions you're interested in
are C ones.

Automation is useful for large and/or evolving libraries which have
conventions. For instance, for glib, the headers all have the same
shapes, they use the same macros and the "object" always goes in the
first argument of the function. PHP's API on the other hand is
everything but consistent.


I think that should cover most of it. As I've said, I haven't updated
cowboy recently. I haven't had much time and I will be horribly until
the end of FOSDEM since I'm presenting there (which is sunday :) ).


Regards,
Adrien Nader

Re: [Caml-list] any automated FFI bindings generators?

[Adrien Nader]

Woops, forgot to link to the website:

https://forge.ocamlcore.org/projects/cowboy/

http://git.ocamlcore.org/cgi-bin/gitweb.cgi?p=cowboy/cowboy.git;a=summary

I also forgot to mention that yacfe-light wasn't neatly
packaged/packageable as far as I know; that's something I have worked on
but not published and I can do so next week.

-- 
Adrien Nader

Re: [Caml-list] any automated FFI bindings generators?

[Markus Weißmann]

On 2014-01-31 02:28, Francois Berenger wrote:
> On 01/30/2014 08:01 PM, Ömer Sinan Ağacan wrote:
>> Hi all,
>>
>> I want to be able to use some very big C libraries from OCaml and I
>> want to automate process of writing bindings as much as possible. 
>> What
>> are my options for this? Do we have any tools to generate bindings?
>
> I am not sure this is what you are looking for, but there is this in 
> OPAM:
>
> ctypes -- Combinators for binding to C libraries without writing any 
> C
>

just tried to use ctypes on an arm-linux system and it failed to 
compile ('[ERROR] "ctypes" is not available for your compiler or your 
OS.');
unfortunately I couldn't find anything regarding the supported 
architectures of ctypes -- any hints on where it works? Only x86/x64?

Regards
Markus

-- 
Markus Weißmann, M.Sc.
Technische Universität München
Institut für Informatik
Boltzmannstr. 3
D-85748 Garching
Germany
http://wwwknoll.in.tum.de/

Re: [Caml-list] any automated FFI bindings generators?

[Daniel Bünzli]

Le vendredi, 31 janvier 2014 à 16:22, Markus Weißmann a écrit :
> just tried to use ctypes on an arm-linux system and it failed to
> compile ('[ERROR] "ctypes" is not available for your compiler or your  
> OS.');
> unfortunately I couldn't find anything regarding the supported  
> architectures of ctypes -- any hints on where it works? Only x86/x64?

No, it works for example on arm-based raspberyy pi. Are you sure you have at least ocaml 4.00.0 and libffi installed ?  

Best,

Daniel

Re: [Caml-list] any automated FFI bindings generators?

[Xavier Leroy]

On 2014-01-30 12:01, Ömer Sinan Ağacan wrote:
> I want to be able to use some very big C libraries from OCaml and I
> want to automate process of writing bindings as much as possible. What
> are my options for this? Do we have any tools to generate bindings?

Yes, several:

- Ctypes   https://github.com/ocamllabs/ocaml-ctypes
- CamlIDL  http://forge.ocamlcore.org/projects/camlidl/
- SWIG     http://www.swig.org/

Enjoy,

- Xavier Leroy

Re: [Caml-list] any automated FFI bindings generators?

[Markus Weissmann]

On 31 Jan 2014, at 16:31 , Daniel Bünzli <daniel.buenzli@erratique.ch> wrote:

> Le vendredi, 31 janvier 2014 à 16:22, Markus Weißmann a écrit :
>> just tried to use ctypes on an arm-linux system and it failed to
>> compile ('[ERROR] "ctypes" is not available for your compiler or your  
>> OS.');
>> unfortunately I couldn't find anything regarding the supported  
>> architectures of ctypes -- any hints on where it works? Only x86/x64?
> 
> No, it works for example on arm-based raspberyy pi. Are you sure you have at least ocaml 4.00.0 and libffi installed ?  
> 

Ah, sorry for the noise then: I only had the Debian-provided 3.12.x and was mislead by the error message into thinking that arm would be the problem!

Best regards
Markus

-- 
Markus Weißmann, M.Sc.
Technische Universität München
Institut für Informatik
Boltzmannstr. 3
D-85748 Garching
Germany
http://wwwknoll.in.tum.de/

Re: [Caml-list] any automated FFI bindings generators?

[Adrien Nader]

Hi,

I've been told over IRC that I made it sound like cowboy is a pet
project that is very young, experimental and/or incomplete.

That is definitely not the case and for the glib-based libraries
support, it binds dozens and dozens of data structures, thousands of
functions and, in the case of webkit-gtk, it generates 12k lines of code
(C, OCaml and lablgtk-specific domain-specific languages).

-- 
Adrien Nader