[Caml-list] create a closure in external C function?

[Caml-list] create a closure in external C function?

[Chris Hecker]

I can't find any docs on this, except for a couple hints that it might be possible...

I want to create a function and return it from an external C function:

type ii = int -> int
val f : int -> ii = "c_function"

where c_function looks something like this:

value c_function( value i )
{
    return Val_closure(&some_c_int_int_function);
}

Is this possible?  The Closure_tag is not very well documented, but from tracing through the asm for some callbacks and passing closures to C functions, it appears that the first word is a pointer to the actual generated code for the caml function.  So, if you wanted to make this work you'd have to return a pointer to a caml function that calls _caml_c_call, which isn't really documented either (and looks platform specific, although it seems to just be jumping to the address of the passed C function after setting things up).

Any hope of getting this working?  Is caml_c_call designed to be a C callable function on all platforms (it doesn't look like it, since it takes a parm in registers on x86, which is unusual)?  Is the bytecode sitatution completely different (I assume so)?

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Fabrice Le Fessant]

What you can do is create the closure in ML, and call a C stub inside
this closure that will call your C function. I tried to implement what
you were asking for (i.e. dlsym in Caml), but the main problem is 
building an arbitrary frame for the C call from the argument types.

An example:

I finally had a C function receiving three arguments. This function is
called from a ML closure used as a stub for the function received by
dlsym. "ml_call_1" is called with the C function as first parameter,
a list of types as second parameter, and the argument as last parameter.

value ml_ccall_1(value f_v, value types_v, value arg1_v)
{

}

My problem is: how to call the function f_v, using the types in
types_v (either integer, char, string or double) to build the call,
and using arg1_v (translated using its type) as argument.

Of course, I can implement this easily on my PC, but I don't know any
way to implement such a function in an architecture-independant way...

- Fabrice

PS: I can give you the code I have already written for building the
closure, and so on if you are interested... but I'm stuck on this
function !
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Xavier Leroy]

> I want to create a function and return it from an external C function:
> 
> type ii = int -> int
> val f : int -> ii = "c_function"
> 
> where c_function looks something like this:
> 
> value c_function( value i )
> {
>     return Val_closure(&some_c_int_int_function);
> }
> 
> Is this possible?

Not directly, at least.  Building closures from C isn't impossible,
just very delicate and highly processor-dependent (in the case of the
native-code compiler).

> So, if you wanted to make this work you'd have to return a pointer
> to a caml function that calls _caml_c_call, which isn't really
> documented either (and looks platform specific, although it seems to
> just be jumping to the address of the passed C function after
> setting things up).

Right. caml_c_call does part of the job, but is indeed very
platform-specific.

If the C functions you're interested in all have the same type,
say "int" to "int", there is an easy solution: instead of returning
them to Caml as Caml functions, return them as values of an abstract
type, and provide a separate "invoke" primitive:

(* Caml code *)

type ii
external ii_invoke: ii -> int -> int = "ii_invoke"
external c_function: ... -> ii = "c_function"

(* C code *)

value c_function(...)
{
  return (value) &some_c_int_int_function;
}

value ii_invoke(value c_function, value arg)
{
  return Val_int(((int (*)(int)) c_function)(Int_val(arg)));
}

Then, on the Caml side, you can always partially apply ii_invoke to a
value of type ii to get back the int->int Caml function you're looking
for.

This solution easily extends to the case of a finite number of C
function types, known in advance.  As Fabrice noted, the hard part is
to deal with arbitrary many C function types, determined dynamically.
There, you'd need some kind of dynamic invocation interface for C
functions, and I don't know of any C library that provides this in a
portable or semi-portable fashion.  (The closest thing that comes to
my mind is run-time code generators such as `C or Dyn-C.)

Hope this helps,

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Chris Hecker]

At 06:01 PM 3/6/01 +0100, Xavier Leroy wrote:
>Not directly, at least.  Building closures from C isn't impossible,
>just very delicate and highly processor-dependent (in the case of the
>native-code compiler).
>Right. caml_c_call does part of the job, but is indeed very
>platform-specific.

It seems like most of the platform specific parts are already there in pieces in the native compiler itself.  I'm not familiar enough with calling conventions on the other platforms, but is it true that one could stitch together a bunch of the asmrun functions to give us a simple c-callable function that can be passed as a closure?  Or, are the caml calling conventions such that you need to do special stuff depending on how many arguments there are?

> As Fabrice noted, the hard part is
>to deal with arbitrary many C function types, determined dynamically.
>There, you'd need some kind of dynamic invocation interface for C
>functions, and I don't know of any C library that provides this in a
>portable or semi-portable fashion.
> ...
At 11:50 AM 3/6/01 +0100, Fabrice Le Fessant wrote:
>My problem is: how to call the function f_v, using the types in
>types_v (either integer, char, string or double) to build the call,
>and using arg1_v (translated using its type) as argument.
>Of course, I can implement this easily on my PC, but I don't know any
>way to implement such a function in an architecture-independant way...

I think I can do this part with the ffcall library.  It has functions for building c-callable functions, both in the stdargs way but with return values as well, and the opposite (where you incrementally tell it what paramters the function takes, and then it gives you back a pointer you can call as a full c function call).  It's highly portable (I think it supports more platforms than ocaml, even, so it's overkill).

I was hoping to avoid the "list of parms" solution and just get both ends hooked up directly, which would be really cool.

>PS: I can give you the code I have already written for building the
>closure, and so on if you are interested... but I'm stuck on this
>function !

Yes, that'd be great.  Please email me the code.

Chris

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Marcin 'Qrczak' Kowalczyk]

Tue, 6 Mar 2001 18:01:28 +0100, Xavier Leroy <Xavier.Leroy@inria.fr> pisze:

> As Fabrice noted, the hard part is to deal with arbitrary many C
> function types, determined dynamically. There, you'd need some kind
> of dynamic invocation interface for C functions, and I don't know
> of any C library that provides this in a portable or semi-portable
> fashion.

What about another issue: converting a function closure to a C
function pointer? This too cannot be done portably, can be done
unportably, is needed in the functional programming context, and
I don't know any C library or compiler extension which provides
it (besides gcc's downward closures, but they are not sufficient).
A generic C interface would probably express it as partial application.

Semi-standard Haskell's foreign function interface provides this
(converting a Haskell's function closure to a C function pointer).
There is a function which frees the data assiociated with a function
pointer produced that way.

Last time I looked at this, I hadn't found this functionality in OCaml.
You can at most register OCaml's functions under textual names. This
interface is too limited. You can't even unregister them.

Should OCaml provide this? I haven't really used OCaml much, but
imagine that currently you have to rely on whatever poor man's closure
mechanism was provided by the C library you are interfacing to.
E.g. gtk+ uses the standard C trick of bundling a callback function
pointer with a void * value which is passed to the function as an
additional argument.

Designers of some other libraries were not aware of the fact that
there exist languages which treat functions as data, e.g. readline
wants just function pointers, and it even doesn't let the programmer
know when the function pointer is no longer needed.

The implementation of this conversion must of course build
heap-allocated machine code at runtime, hiding the data pointer
inside it.

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^                      SYGNATURA ZASTĘPCZA
QRCZAK

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Chris Hecker]

>Last time I looked at this, I hadn't found this functionality in OCaml.
>You can at most register OCaml's functions under textual names. This
>interface is too limited. You can't even unregister them.

You can pass a closure to a C function and apply it to arguments with the callback.h library pretty easily.  Or were you talking about something else?

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Marcin 'Qrczak' Kowalczyk]

Tue, 06 Mar 2001 11:13:24 -0800, Chris Hecker <checker@d6.com> pisze:

> You can pass a closure to a C function and apply it to arguments
> with the callback.h library pretty easily.

It's not enough to convert a closure to a C function pointer.
For example to wrap C's qsort() in this interface:
    ('a -> 'a -> int) -> 'a array -> unit

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^                      SYGNATURA ZASTĘPCZA
QRCZAK

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Chris Hecker]

>> You can pass a closure to a C function and apply it to arguments
>> with the callback.h library pretty easily.
>It's not enough to convert a closure to a C function pointer.
>For example to wrap C's qsort() in this interface:
>    ('a -> 'a -> int) -> 'a array -> unit

Right, but it's part of the solution.  The plan is to make it so you can do that automatically.  On the C side, we'll use the ffcall trampoline/vacall interface to pass qsort the pointer to a function that calls back the caml function with the right parms.  It's gonna be beautiful (or go down in flames, one of the two :).  There are only a couple loose ends to tie up and I'm confident we'll be able to get it working in its full insane generality.

You'll still want to use an interface generator for any permanent functions you want to hook because this is going to generate a bunch of code to loop over arguments at runtime and whatnot, but for dynalinking to dlls (or doing stuff like OpenGL extensions) it'll rule.  Hopefully it'll only be 20% or so slower than a direct external function call.

It's not going to handle records or polymorphism at first, of course (or even passing or returning closures when I first write it).  I've got a plan for handling records, but I need to understand polymorphism more before I can think about that (in fact, I can't do any real work on this for a couple weeks, but I'm trying to lay the groundwork).

I'm also a little terrified of making this play nice with the GC, but we'll blow up that bridge when we come to it.

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Marcin 'Qrczak' Kowalczyk]

Tue, 06 Mar 2001 14:56:51 -0800, Chris Hecker <checker@d6.com> pisze:

> On the C side, we'll use the ffcall trampoline/vacall interface
> to pass qsort the pointer to a function that calls back the caml
> function with the right parms.

I've just seen ffcall. callback is approximately what I wanted.

trampoline+vacall is evil. It uses a global variable; this is not
thread safe.

callback seems to be better, but still more kludgy than I expected.
I'm not sure if it's the right choice for OCaml; it should not be
*that* ugly. Especially if you statically know types of arguments,
which is the case I am talking about. Unfortunately it's very
processor-dependent.

I would like to have this functionality in OCaml's foreign function
interface, without forcing to rely on an inconvenient third-party
package.

I'm not sure how it should look like, because OCaml has a different
approach to FFI than Haskell: the OCaml<->C glue is in C, where for
Haskell<->C the glue is in Haskell. So Haskell defines a mapping
from Haskell function types involving primitive types to C types,
and a special kind of declaration which defines converters from
Haskell functions to C function pointers of particular types. The
OCaml's approach would need to call the conversion from the C level,
in which case the compiler has no chance of generating stubs tailored
for a particular type but must offer a generic C interface with its
own way to describe C types.

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^                      SYGNATURA ZASTĘPCZA
QRCZAK

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Chris Hecker]

>trampoline+vacall is evil. It uses a global variable; this is not
>thread safe.

Yeah, I was going to look at that closer to see if there's any way to make it thread safe.  It seems a shame to have all of that cross platform work done and then shove the thing in a global...

> Especially if you statically know types of arguments,
>which is the case I am talking about.

Which case are you talking about?  Are you working on this as well?  Are you just talking about wanting to expose qsort and other functions that take callbacks?

> I would like to have this functionality in OCaml's foreign function
>interface, without forcing to rely on an inconvenient third-party
>package.

That would be great, but I'm not sure if anybody's working on this at INRIA.  Heck, we might as well do a proof of concept first before they waste their time on it anyway, and then if it's useful we'll see about making it more official.  If it was built into the compiler it could be made super-clean, basically like external is now...I can see it now...extern[al] "C"!  <ducking and running>  

My original plan wasn't to do the whole thing, it was just to get wglGetProcAddress (and the glx version) working for OpenGL for a library Kipton Barros and I are going to be working on, and then I realized that if I got that working it wouldn't be much harder to just generalize it to GetProcAddress/dlsym, which would have the potential to be hugely useful for all caml programmers, not just OpenGL programmers.  So, I'll probably just do that, assuming it turns out to be possible.

Chris


-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Fergus Henderson]

On 06-Mar-2001, Chris Hecker <checker@d6.com> wrote:
> 
> I think I can do this part with the ffcall library.  It has functions for
> building c-callable functions, both in the stdargs way but with return values
> as well, and the opposite (where you incrementally tell it what paramters the
> function takes, and then it gives you back a pointer you can call as a full c
> function call).  It's highly portable (I think it supports more platforms
> than ocaml, even, so it's overkill).

ffcall is not very portable at all, but just widely *ported*.
Note the difference!

-- 
Fergus Henderson <fjh@cs.mu.oz.au>  |  "I have always known that the pursuit
                                    |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr

Re: [Caml-list] create a closure in external C function?

[Marcin 'Qrczak' Kowalczyk]

Tue, 06 Mar 2001 16:44:06 -0800, Chris Hecker <checker@d6.com> pisze:

> > Especially if you statically know types of arguments,
> > which is the case I am talking about.
> 
> Which case are you talking about?  Are you working on this as well?

Not quite. But if I ever write an OCaml binding to a C library,
I might need it; and other people might similarly need it.
It's a concrete functionality which is missing from OCaml but
makes sense. I use its equivalent in Haskell to wrap C libraries
which use callbacks in Haskell's higher order functions.

> Are you just talking about wanting to expose qsort and other
> functions that take callbacks?

Yes.

-- 
 __("<  Marcin Kowalczyk * qrczak@knm.org.pl http://qrczak.ids.net.pl/
 \__/
  ^^                      SYGNATURA ZASTĘPCZA
QRCZAK

-------------------
To unsubscribe, mail caml-list-request@inria.fr.  Archives: http://caml.inria.fr