[Caml-list] dynamically loading C functions

[Caml-list] dynamically loading C functions

[Chris Hecker]

I'm looking into dynamically linking to C functions "mostly" type safely, and I've run into a couple small snags:  

1.  I need to generate the thunk (or trampoline, it's sometimes called) to convert the caml values to their C values.  Assume I've found some way (see below) to tell the library what function I'm linking to, and what it's type it is supposed to be.  I need to generate a thunk (at runtime) that converts the caml versions of the parameters to the C versions.

For example, say I've got this:

(* int TheFunction( double, double ); ... ignore how I'm typing 
   get_function for until below *)
let f = get_function "TheFunction" "float -> float -> int"

So, if I'm going to be able to call f from caml, get_function is going to need to generate a stub that takes two values and returns a value, and converts those to their C types with the mlvalues.h macros before calling TheFunction (and handles its return type).

This piece of magic doesn't have to be very smart, since it's going to be predefined chunks of conversion code, but it will have to be generated on the fly by piecing together these chunks if I'm really going to allow completely dynamic linking.

This is obviously not very cross platform.  There is a library called ffcall which has a thing called avcall in it that looks like it might do what I want with some massaging, and it's cross platform.   

I think I also need to worry about generating the function that gets called directly by the return of get_function, but I think I can use the ffcall vacall library for that.

Does anybody have any experience with avcall, vacall, or ffcall?  Is there a better way to do this?


2.  I want to have get_function return a correctly typed function that can just be called with the correct caml types.  So, in the example above, f should be float -> float -> int.  There are two parts to this problem:  I need to have the source code specify the type of the linked function to the link library so it can generate the thunk as above, and I need to specify the type of the return function to the caml compiler.

I think I have a few options:

- I can try to use the printf formatting stuff and try to get this to do the right thing, so you'd pass a format string to the function.  This has problems with return values, since those aren't read from the format string.  I can do

val get_function: ('a,unit,'c) format -> 'c -> 'a

and that will work, but it requires me to pass in a dummy parameter for 'c to get it to parameterize 'a correctly on return type.  Is there a good way to specify types for this sort of thing?  Maybe use the format string for the parms and some kind of explicit specification for the return type?  But get_function needs to return 'a as its return type...

- I can just have val get_function: string -> 'a, and have it parse the string internally to get the type, and then let 'a be type inferred in the caml source from the use of the return value.  Is that a good idea?  It seems like that might make for ambiguities if you ignore return values or curry it.

- I can tell people to "cast" the 'a return type, so you'd do this:

let f = (get_function "TheFunction: float -> float -> int" : float -> float -> int)

but, this requires the error prone double-specification of the type, once to the library, and once to the source code.

It seems to me that the printf format string one is "cleanest" and type-safest, but the return value thing is a hack.  I guess I could also patch the compiler and extend the format string parser to take a special % token to specify a return type, but that would be a hack itself and a lot of work.

Any ideas?  And yes, this all goes away if I have a preprocessing step that generats the caml/c interface, but I'd really like to be able to do this at runtime.

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Marcin 'Qrczak' Kowalczyk]

Mon, 05 Mar 2001 12:28:26 -0800, Chris Hecker <checker@d6.com> pisze:

> - I can tell people to "cast" the 'a return type, so you'd do this:
> 
> let f = (get_function "TheFunction: float -> float -> int" : float -> float -> int)
> 
> but, this requires the error prone double-specification of the type,

You can provide combinators to build the type information:

val int_t   : int t
val float_t : float t
val (@->)   : 'a t -> 'b t -> ('a -> 'b) t
(* Begins with @ so it's right-associative :-) *)

val get_function : string -> 'a t -> 'a

let f = get_function "TheFunction" (float_t @-> float_t @-> int_t)

(I just imagined the idea in Haskell and translated implicit Haskell's type
classes into explicit OCaml's dictionary passing.)

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^                      SYGNATURA ZASTĘPCZA
QRCZAK

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Chris Hecker]

>You can provide combinators to build the type information:

Wow, that's pretty cool.  I'm not that familiar with abstract types and modules in caml (or any functional language), so this was kind of an eye opener.  I think I understand how they work a bit more now, thanks.  I do have one thing I can't figure out.  Here's my test module:

module C:
    sig
      type 'a t
      val int_t   : int t
      val float_t : float t
      (* Begins with @ so it's right-associative :-) *)
      val (@->)   : 'a t -> 'b t -> ('a -> 'b) t
      val get_function : string -> 'a t -> 'a

      val conv : 'a t -> int list
    end =
  struct
    type 'a t = T of int | Tl of int list
    let int_t = T 1
    let float_t = T 2
    let (@->) a b = match a,b with 
      (T a,T b) -> Tl [a;b]
    | (T a,Tl b) -> Tl (a :: b)
    | (Tl a,Tl b) -> Tl (a @ b)
    | (Tl a,T b) -> Tl (a @ [b])
    let get_function s l = Obj.magic s (* just get it to compile *)
        
    let conv (Tl a) = a
  end

The way I've done 'a t is as a variant int list.  This works, but my @-> flattens nested lists because of the pattern match, so while the type system differentiates between

# (float_t @-> int_t @-> float_t @-> float_t @-> int_t);;
- : (float -> int -> float -> float -> int) C.t = <abstr>

and (note the nested parens):

# (float_t @-> (int_t @-> float_t @-> float_t) @-> int_t);;
- : (float -> (int -> float -> float) -> int) C.t = <abstr>

my internal representation doesn't:

# conv (float_t @-> int_t @-> float_t @-> float_t @-> int_t);;
- : int list = [2; 1; 2; 2; 1]

# conv (float_t @-> (int_t @-> float_t @-> float_t) @-> int_t);;
- : int list = [2; 1; 2; 2; 1]

I thought I could fix this by saying type 'a t = T of int | Tl of 'a t list, but that seems to constrain (@->) to  be 'a t -> 'a t -> 'a t and I lose the cruicial ('a -> 'b) t combinator (not sure if I'm using that word correctly)  and I get a type error.  Any idea how I could get it to nest?

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Chris Hecker]

>I thought I could fix this by saying type 'a t = T of int | Tl of 'a t list, but that seems to constrain (@->) to  be 'a t -> 'a t -> 'a t and I lose the cruicial ('a -> 'b) t combinator (not sure if I'm using that word correctly)  and I get a type error.  Any idea how I could get it to nest?

Okay, I fixed this by introducing another type that didn't need to be parameterized (see below).  I guess I should have thought of that earlier.  Now my only problem is that I collapse int_t -> (int_t -> int_t) to int_t -> int_t -> int_t, which is correct for languages with currying, but C doesn't curry, so I need to keep those parens around to generate the right type.  Not sure how to do this, but I might be able to pull something fancy...gotta think about it.  Or, maybe I should left associate, which will make the return type last always, rather than first, and any time I come across a list on the right I should preserve it...

Chris

(* handles nested function types *)

module C:
    sig
      type 'a t
      val int_t   : int t
      val float_t : float t
      (* Begins with @ so it's right-associative :-) *)
      val (@->)   : 'a t -> 'b t -> ('a -> 'b) t
      val get_function : string -> 'a t -> 'a

      val print : 'a t -> unit
    end =
  struct
    type tt = T of int | Tl of tt list
    type 'a t = tt
    let int_t = T 1
    let float_t = T 2
    let (@->) a b = match a,b with 
      ((T _ as a),(T _ as b)) -> Tl [a;b]
    | ((T _ as a),Tl b) -> Tl (a :: b)
    | (((Tl _) as a),((Tl _) as b)) -> Tl [a;b]
    | (((Tl _) as a),(T _ as b)) -> Tl [a;b]
    let get_function s l = Obj.magic s (* just get it to compile *)
        
    let print el =
      let rec doprn el =
        match el with
          T a -> Printf.printf " %d " a
        | Tl a -> begin print_string " [ "; List.iter ~f:doprn a; print_string " ] " end
      in doprn el      
  end

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Marcin 'Qrczak' Kowalczyk]

Mon, 05 Mar 2001 16:55:25 -0800, Chris Hecker <checker@d6.com> pisze:

> Now my only problem is that I collapse int_t -> (int_t -> int_t)
> to int_t -> int_t -> int_t, which is correct for languages with
> currying, but C doesn't curry, so I need to keep those parens
> around to generate the right type.

Do you indeed support returning functions from C? OK, so you need to
distinguish more than the OCaml type tells. For example by having two
functions of type 'a t -> 'b t -> ('a -> 'b) t: one to use between
arguments and another for the final result.

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^                      SYGNATURA ZASTĘPCZA
QRCZAK

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Chris Hecker]

>Do you indeed support returning functions from C? OK, so you need to
>distinguish more than the OCaml type tells. For example by having two
>functions of type 'a t -> 'b t -> ('a -> 'b) t: one to use between
>arguments and another for the final result.

Right, I thought of that, and I also thought of having a special token in the "type stream" that indicates a return value.  Neither of these are quite as natural as I'd like, though.  I think having a @@-> that you use between args and return isn't terrible, but it's just even more syntax to make this harder to use and more error prone.  :)  The get_function could raise an exception if every function in the specification didn't have a return type specifed, though.

I've got it wired up to use the Bigarray ('a,'b) kind stuff now, so the caml function and the c function type signatures parallel each other as ('aa,'ab) t -> ('ba,'bb) t -> ('aa -> 'ba,'ab -> 'bb) t (and I can reuse the Bigarray type hierarchy, although I don't know how to get access to the internal definitions of the Bigarray abstract types which I need to actually return the information, I think...or maybe not, so I may need to copy them, which sucks...sometimes having "#include<>" is useful  ;).

As for whether I need to support returning functions, all of this is relative.  I would really like to just solve the entire problem at once, if possible.  I'd like to make this work for generic dynalinking, but the very case I was going to use first (wglGetProcAddress) does actually return a function pointer.  :)

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Fabrice Le Fessant]

I've written a Dlopen module for x86 and alpha available at
http://pauillac.inria.fr/~lefessan/src/dlopen.tar.gz

It allows this kind of thing:

 cremant:~/devel/dlopen%  ./top   
        Objective Caml version 3.00+9 (2000-07-08)

# open Dlopen;; 
# let dll = dlopen "./test.so"  [RTLD_NOW];;
val dll : Dlopen.dll = <abstr>
# let add_int = dlsym dll "add_int" "%d:%d%d";;
val add_int : int -> int -> int = <fun>
# let print_string = dlsym dll "print_string" "%d:%s";;
val print_string : string -> int = <fun>
# let add_double = dlsym dll "add_double" "%f:%f%f";;
val add_double : float -> float -> float = <fun>
# add_double (1.0) (2.0);;
x: 1.000000 y: 2.000000
- : float = 3.000000
# print_string "bonjour";;
bonjour
- : int = 10
# add_int 1 2;;
x: 1 y: 2
- : int = 3
# 

where test.c is:

#include <stdio.h>
#include <stdlib.h>

int add_int(int x, int y)
{  printf("x: %d y: %d\n",x,y); return x+y; }

int print_string(char *s)
{  printf("%s\n", s);  return 10;}

double add_double(double x, double y)
{ printf("x: %f y: %f\n",x,y); return x+y;}

- Fabrice
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Chris Hecker]

>I've written a Dlopen module for x86 and alpha available at
>http://pauillac.inria.fr/~lefessan/src/dlopen.tar.gz

Neat!  This is a very nice type hack to get the return value parameterizable with format:

val dlsym : dll -> string -> ('c -> 'a,unit, 'c) format -> 'a

When I was thinking of using the printf format strings, I couldn't figure out how to get the implicit 'c return value out of there on the result, but putting it in the first type argument as well is ingenious!

Does this library ever build closures around C functions and pass them back to caml (like you mentioned in a previous mail)?  I don't see the code that does that anywhere.

 From looking at the cmmgen.ml source, it looks like closures are a pretty simple layout (there's a header before these):

if physical arity 1:
word: pointer to function
word: 3 (arity (as caml int 1 lsl 1 + 1))

if physical arity > 1:
word: pointer to a curry function generated at link time
word: arity
word: pointer to function

But I've only looked at x86 so far.  The compiler actually is pretty smart (as if we didn't know that already :).  From what I can tell, if a module has been compiled, the cmx file (and maybe the cmi file) has extra information in it about the physical arity of its functions, so when they're called from another module, the compiler can just directly call the function rather than grovel around in its closure datastructure to find its physical arity (or call a subroutine to do this, which is what happens if you haven't compiled a module and you call a function in it, or just have an mli file but no ml file backing it).  I guess this is one of the currying optimizations people have been talking about.  I wonder how compilation-unit-order-dependent this is...

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] dynamically loading C functions

[Fabrice Le Fessant]

I've updated the dlopen ocaml port:
http://pauillac.inria.fr/~lefessan/src/dlopen/

It now works on 4 systems: x86, alpha, sparc and PPC.

It can use either printf formats or type combinators to specify
returned functions types. 

Type combinators must be use if you want to pass structs to C
functions (encoded by tuples in Caml), or get side-effects.

For example, it test.c defines:

struct tt {
  char field1;
  int  field2;
  char field3;
  double field4;
};

int set_tuple(struct tt* t)
{
  printf("%c - %d - %c - %f\n", t->field1, t->field2, t->field3, t->field4);
  t->field2 = 24;
  return 0;
}

You can use the following code:

# open Dlopen;; 
# let dll = dlopen "./test.so"  [RTLD_NOW];;
val dll : Dlopen.dll = <abstr>

# let struct_t =  record4_t (char_t, mutable_t int_t, char_t, float_t);;
val struct_t : (char * int ref * char * float) Dlopen.meta = <abstr>

# let set_tuple = dlsym_t dll "set_tuple" (
    star_t (mutable_t struct_t) @-> int_t);;
val set_tuple : (char * int ref * char * float) ref option -> int = <fun>

# let x =ref 1;;
val x : int ref = {contents=1}
# let y = ref ('a',x,'b',2.0);;
val y : (char * int ref * char * float) ref =
  {contents='a', {contents=1}, 'b', 2}

# set_tuple (Some y);;
a - 1 - b - 2.000000
- : int = 0

# x,y;;
- : int ref * (char * int ref * char * float) ref =
{contents=24}, {contents='a', {contents=24}, 'b', 2}

where both the references x and y capture the side-effect of the
function on the record (note that sharing is not preserved, ie in the
final result, 
   {contents=24}, {contents='a', {contents=24}, 'b', 2}
       ^                                ^
       ^-------------- != --------------^

Be careful to exactly encode the C arguments format in the Caml type
combinators (in particular, pointers are encoded by star_t as in the
example).

Regards,

- Fabrice
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr