Wrapping var_args, or C ... in ocaml?

Wrapping var_args, or C ... in ocaml?

[Guillaume Yziquel]

Hello.

I'm currently looking at:

	http://docs.python.org/c-api/arg.html

and I would like to know how to wrap up C functions with va_list of with 
an ellipsis. Is this documented somewhere, or has someone already done 
something like this?

-- 
      Guillaume Yziquel
http://yziquel.homelinux.org/

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Richard Jones]

On Sun, Feb 14, 2010 at 04:46:20PM +0100, Guillaume Yziquel wrote:
> Hello.
> 
> I'm currently looking at:
> 
> 	http://docs.python.org/c-api/arg.html
> 
> and I would like to know how to wrap up C functions with va_list of with 
> an ellipsis. Is this documented somewhere, or has someone already done 
> something like this?

It really depends on the function and how it will be used.  It
might translate to any of:

(1) A collection of functions implementing different aspects of the C
function.  eg. The open(2) function in Unix is really a varargs
function, and depending on whether you want to open a file for input,
output, create, etc. you'd probably be better off with different
functions in OCaml.  (Unix.openfile does _not_ do this ...)

(2) A simple list, eg. for a C function that takes a NULL-terminated
list of strings.

(3) A variant list of variants, or option labels, eg. for a C function
that takes 'type, value'(s), such TIFFSetField in
libtiff. (http://www.remotesensing.org/libtiff/man/TIFFSetField.3tiff.html)

(4) Something very specialized, eg. the 'printw' function in ncurses
is like printf and so would need quite a tricky implementation in
OCaml.  (Probably best to use ksprintf to convert to a string in OCaml
and then pass printw ("%s", str) in C).

In libguestfs where we autogenerate bindings we avoided varargs
altogether, because it's hard to map such a concept to all the
different languages we support.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Richard Jones]

On Sun, Feb 14, 2010 at 06:06:50PM +0000, Richard Jones wrote:
> (3) A variant list of variants, or option labels

A comma got dropped there somehow; it should be:

  A variant, list of variants, or option labels ...

which hopefully makes more sense.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Guillaume Yziquel]

Richard Jones a écrit :
> On Sun, Feb 14, 2010 at 04:46:20PM +0100, Guillaume Yziquel wrote:
>> Hello.
>>
>> I'm currently looking at:
>>
>> 	http://docs.python.org/c-api/arg.html
>>
>> and I would like to know how to wrap up C functions with va_list of with 
>> an ellipsis. Is this documented somewhere, or has someone already done 
>> something like this?
> 
> It really depends on the function and how it will be used.  It
> might translate to any of:
> 
> (1) A collection of functions implementing different aspects of the C
> function.  eg. The open(2) function in Unix is really a varargs
> function, and depending on whether you want to open a file for input,
> output, create, etc. you'd probably be better off with different
> functions in OCaml.  (Unix.openfile does _not_ do this ...)

Not the case.

> (2) A simple list, eg. for a C function that takes a NULL-terminated
> list of strings.

Could be.

> (3) A variant list of variants, or option labels, eg. for a C function
> that takes 'type, value'(s), such TIFFSetField in
> libtiff. (http://www.remotesensing.org/libtiff/man/TIFFSetField.3tiff.html)

No.

> (4) Something very specialized, eg. the 'printw' function in ncurses
> is like printf and so would need quite a tricky implementation in
> OCaml.  (Probably best to use ksprintf to convert to a string in OCaml
> and then pass printw ("%s", str) in C).

I do not think so.

> In libguestfs where we autogenerate bindings we avoided varargs
> altogether, because it's hard to map such a concept to all the
> different languages we support.

True.

But, I mean, from the point of view of the ABI, there's not much 
trickery in the concept. It looks that it is C that is not mapping the 
concept to its fullest potential.

I mean, it seems that varargs means on the receiving end "the number of 
arguments you'r giving me, as a function, is not limited", whereas on 
the sending end, you hard-code the number of arguments in your C code.

Is there a way to map an OCaml list to an ellipsis? Or is it a C limitation?

-- 
      Guillaume Yziquel
http://yziquel.homelinux.org/

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Richard Jones]

On Sun, Feb 14, 2010 at 11:46:10PM +0100, Guillaume Yziquel wrote:
> Not the case.
[etc]

It would help if you were to be more specific about the function that
you're trying to bind.

> But, I mean, from the point of view of the ABI, there's not much 
> trickery in the concept. It looks that it is C that is not mapping the 
> concept to its fullest potential.
> 
> I mean, it seems that varargs means on the receiving end "the number of 
> arguments you'r giving me, as a function, is not limited", whereas on 
> the sending end, you hard-code the number of arguments in your C code.

In the C case the sender pushes arguments right to left on the stack,
and the receiver must deduce in an unspecified way how many arguments
were pushed.  This can be because the sender promises to
NULL-terminate the list, or encodes in an earlier argument some
indication of how many arguments follow, or they might even have given
it in a previous function call, or have just agreed it in the
documentation.

If the receiver gets it wrong, there is plenty of scope for
catastrophic errors to occur -- and this is not merely a theoretical
problem, but a very real problem that many C programmers have
encountered.  So I'm quite happy that OCaml doesn't make this kind of
poorly-specified function easy to implement.

> Is there a way to map an OCaml list to an ellipsis? Or is it a C
> limitation?

I'm not sure exactly what this means.  An OCaml list is a
well-defined, well-typed version of the C ellipsis isn't it?

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Guillaume Yziquel]

Richard Jones a écrit :
> On Sun, Feb 14, 2010 at 11:46:10PM +0100, Guillaume Yziquel wrote:
>> Not the case.
> [etc]
> 
> It would help if you were to be more specific about the function that
> you're trying to bind.

http://docs.python.org/c-api/arg.html

There's quite a few there...

-- 
      Guillaume Yziquel
http://yziquel.homelinux.org/

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Richard Jones]

On Mon, Feb 15, 2010 at 12:13:17AM +0100, Guillaume Yziquel wrote:
> Richard Jones a écrit :
> >On Sun, Feb 14, 2010 at 11:46:10PM +0100, Guillaume Yziquel wrote:
> >>Not the case.
> >[etc]
> >
> >It would help if you were to be more specific about the function that
> >you're trying to bind.
> 
> http://docs.python.org/c-api/arg.html
> 
> There's quite a few there...

OK ... I've done a lot of bindings of C functions to Python and
obviously the functions you've pointed out above are important for
that.

However I'm still confused what you are trying to do here.  If you're
trying to bind the above, maybe look first at PyCaml?

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Richard Jones]

As an example of a Python binding of the sort we generate in
libguestfs (using OCaml code to generate it :-)

  http://www.annexia.org/tmp/guestfs-py.c.txt
  [Search for calls to PyArg_ParseTuple]

The Python binding technique is sort of interesting.  On the other
hand, Python is decoding that kind-of-C-format-string arg to
PyArg_ParseTuple entirely at runtime which makes it really slow (but
not the slowest thing in Python by any means -- that language takes
being inefficient to a new level).

Out of all the language bindings that we support, the one with the
most natural FFI to C [if you exclude C++] is C#.  In C# you can
express C structures, C calling conventions and so on directly in the
language, and it is very well documented how to do this.  This makes
C# calling C shared libraries / DLLs very natural.

  http://www.annexia.org/tmp/Libguestfs.cs.txt

The worst of all of them is Haskell.  Not because the Haskell FFI is
bad, but because it's (a) obscure and undocumented and (b) the only
one of the programming languages apart from C# where you aren't
basically writing C code.  If you don't already know Haskell, it's
very difficult to writing bindings for Haskell.

  http://www.annexia.org/tmp/Guestfs.hs.txt

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Guillaume Yziquel]

Richard Jones a écrit :
> On Mon, Feb 15, 2010 at 12:13:17AM +0100, Guillaume Yziquel wrote:
>> Richard Jones a écrit :
>>> On Sun, Feb 14, 2010 at 11:46:10PM +0100, Guillaume Yziquel wrote:
>
> However I'm still confused what you are trying to do here.  If you're
> trying to bind the above, maybe look first at PyCaml?

Well, somehow, PyCaml seems to never have wanted to work with me.

I tried Art Yerkes' version, Thomas Fishbacher's version, Henrik 
Stuart's version, and Yoann Padioleau's merge of these versions. Somehow 
things always seem wrong. I get a segfault with Yoann Padioleau's 
version due to the layout of the Python table function (WTF!? BTW).

So I'm rewriting a Python embedding.

http://yziquel.homelinux.org/gitweb/?p=ocaml-python.git;a=shortlog;h=refs/heads/yziquel
http://yziquel.homelinux.org/debian/pool/main/o/ocaml-python/
http://yziquel.homelinux.org/topos/api/ocaml-python/Python.html

Sample session:

> yziquel@seldon:~$ ocaml
>         Objective Caml version 3.11.1
> 
> # #use "topfind";;
> - : unit = ()
> Findlib has been successfully loaded. Additional directives:
>   #require "package";;      to load a package
>   #list;;                   to list the available packages
>   #camlp4o;;                to load camlp4 (standard syntax)
>   #camlp4r;;                to load camlp4 (revised syntax)
>   #predicates "p,q,...";;   to set these predicates
>   Topfind.reset();;         to force that packages will be reloaded
>   #thread;;                 to enable threads
> 
> - : unit = ()
> # #require "python.interpreter";;
> /usr/lib/ocaml/python: added to search path
> /usr/lib/ocaml/python/python.cma: loaded
> /usr/lib/ocaml/python/oCamlPython.cmo: loaded
> # open Python;;
> # let dolfin = Module.import "dolfin";;
> val dolfin : Python.pymodule Python.t = <abstr>
> # let dictionary = Module.get_dict dolfin;;
> val dictionary : Python.dict Python.t = <abstr>
> # let keys = Dict.keys dictionary;;
> val keys : string list Python.t = <abstr>
> # let key_list = list_from keys;;
> val key_list : string Python.t list =
>   [<abstr>; <abstr>; <abstr>; <abstr>; <abstr>; <abstr>; <abstr>; <abstr>;
>    <abstr>; <abstr>; <abstr>; <abstr>; <abstr>; <abstr>; <abstr>; <abstr>;
>    <abstr>; <abstr>; <abstr>; <abstr>; <abstr>; ...]
> # let string_list = List.map string_from key_list;;
> val string_list : string list =
>   ["restriction"; "gt"; "precedence"; "TrialFunctions"; "ale"; "as_tensor";
>    "cross"; "__path__"; "shape"; "PETScFactory"; "has_mpi"; "down_cast";
>    "differentiation"; "Mesh"; "has_scotch"; "rot"; "has_slepc"; "DOLFIN_PI";
>    "begin"; "le"; "outer"; "VectorElement"; "parameters"; "ln";
>    "uBLASVector"; "uBLASDenseMatrix"; "tr"; "Assembler"; "terminal";
>    "UnitCube"; "lt"; "CRITICAL"; "hermite"; "derivative"; "logger";
>    "uBLASDenseFactory"; "norm"; "MPI"; "info"; "triangle"; "R1"; "R2";
-- 
      Guillaume Yziquel
http://yziquel.homelinux.org/

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Thomas Fischbacher]

Guillaume Yziquel wrote:

> Richard Jones a écrit :
>> On Mon, Feb 15, 2010 at 12:13:17AM +0100, Guillaume Yziquel wrote:
>>> Richard Jones a écrit :
>>>> On Sun, Feb 14, 2010 at 11:46:10PM +0100, Guillaume Yziquel wrote:
>> However I'm still confused what you are trying to do here.  If you're
>> trying to bind the above, maybe look first at PyCaml?
> 
> Well, somehow, PyCaml seems to never have wanted to work with me.
> 
> I tried Art Yerkes' version, Thomas Fishbacher's version, Henrik 
> Stuart's version, and Yoann Padioleau's merge of these versions. Somehow 
> things always seem wrong. I get a segfault with Yoann Padioleau's 
> version due to the layout of the Python table function (WTF!? BTW).
> 
> So I'm rewriting a Python embedding.

...which is a good idea.

Our version of pycaml (i.e. the one included in the nsim package) works
in the sense that it does not have memory management bugs, as the one in
debian does. However, I think it is not thread-safe from the Python
side. Still, it is based on Art Yerkes' old code. I consider a clean
re-write a very good idea. Would do it myself, if I had the time...

-- 
best regards,
Thomas Fischbacher
t.fischbacher@soton.ac.uk

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Florent Monnier]

Le dimanche 14 février 2010 23:46:10, Guillaume Yziquel a écrit :
> I mean, it seems that varargs means on the receiving end "the number of 
> arguments you'r giving me, as a function, is not limited", whereas on 
> the sending end, you hard-code the number of arguments in your C code.
> 
> Is there a way to map an OCaml list to an ellipsis? Or is it a C
>  limitation?

Yes, as long as I know, for this you should use these kind of tools:

http://sourceware.org/libffi/
http://www.gnu.org/software/libffcall/
http://www.nongnu.org/cinvoke/
http://dyncall.org/


-- 

Re: [Caml-list] Wrapping var_args, or C ... in ocaml?

[Guillaume Yziquel]

Guillaume Yziquel a écrit :
> Florent Monnier a écrit :
>> Le dimanche 14 février 2010 23:46:10, Guillaume Yziquel a écrit :
>>> I mean, it seems that varargs means on the receiving end "the number 
>>> of arguments you'r giving me, as a function, is not limited", whereas 
>>> on the sending end, you hard-code the number of arguments in your C 
>>> code.
>>>
>>> Is there a way to map an OCaml list to an ellipsis? Or is it a C 
>>> limitation?
>>
>> Yes, as long as I know, for this you should use these kind of tools:
>> http://sourceware.org/libffi/
>> http://www.gnu.org/software/libffcall/avcall.html
>> http://www.nongnu.org/cinvoke/
> 
> Phew! These tools are quite tough to handle! (I just tried to use 
> avcall.h on this ellipsis.)

Got it.

The correct code is:

> CAMLprim value ocamlpython_py_tuple_pack (value ml_len, value ml_pyobjs) {
> 
>   av_alist argList;
>   PyObject * retVal;
>   av_start_ptr(argList, &PyTuple_Pack, PyObject*, &retVal);
> 
>   #if defined(__s390__) || defined(__hppa__) || defined(__cris__)
>   #define av_Py_ssize_t av_long
>   #else
>   #define av_Py_ssize_t av_int
>   #endif
> 
>   av_Py_ssize_t(argList, Pyoffset_val(ml_len));
>   while (ml_pyobjs != Val_emptylist) {
>     av_ptr(argList, PyObject*, Pyobj_val(Field(ml_pyobjs, 0)));
>     ml_pyobjs = Field(ml_pyobjs, 1);
>   }
> 
>   av_call(argList);
>   return(Val_owned_pyobj(retVal));
> }

Hope that will be useful to people wishing to wrap up varargs (though 
it's probably a bad idea, since the stack is limited...)

Python function calls are now possible by constructing the argument tuple.

-- 
      Guillaume Yziquel
http://yziquel.homelinux.org/