[Caml-list] Automatic wrapper generator

[Caml-list] Automatic wrapper generator

[skaller]

This is a brief status report concerning the Felix
automatic wrapper generator (reason for posting to Caml list
explained below).

With some platform specific hackery, I am now able to 
wrap 90% of all headers in '/usr/include' and execute
a single Felix test case:

-----------------------
include "stdio_h";
C_hack::ignore(stdio_h::printf(c"%s\n", c"Hello World"));
--------------------

Line 1 includes the wrapper generated for stdio.h.
Line 2 invokes the wrapper for a popular function 'printf()'.
This program compiles and executes correctly.

C_hack::ignore is the same as Ocaml's ignore function,
it throws away the result of calling printf. 
The lexical elements c".." generate C char arrays
(instead of Felix strings which are C++ STL strings).
The include directive pulls in a precompiled header
containing a module named stdio_h, which is used
as a qualifier to the function name printf().

It isn't clear I have got everything right yet,
there is still a long way to go. However I do consider
this a reasonable "proof-of-principle" demonstration
that automatic wrapping of the complete C development
environment is possible. The wrapper generator uses the
frontc/Cil parser, and generates the wrappers for 
of /usr/include in a couple of seconds.

RELEVANCE TO OCAML.
------------------

There isn't much that I'm doing for Felix wrappers
that can't be applied to Ocaml: perhaps with some
extra work it seems possible to make an Ocaml back end
to generate Foreign Function Interfaces for the whole
C environment in a few seconds.

Felix does have a few advantages over Ocaml that may
need to be considered in adapting the code, for example,
in Felix everything is automatically mutually recursive,
and so the order of definitions of types and functions
is irrelevant. Ocaml sometimes needs sequential definitions,
and has some problems with intermodule recursion.

However, I do not see any major obstacles.
[EG: intermodule recursion is a mess to handle by hand...
but it's no problem for a mechanical wrapper generator]

Note of course the generated wrappers are,
at least currently, LOW LEVEL and thus not safe.

for some time, people have been crying for 'MORE LIBRARIES'
for Ocaml.

Here's your chance to get THE LOT in one go.

I'm willing to work on adapting the wrapper generator
so it can be used to generate Ocaml wrappers, 
provided it continues to be useful generating
Felix ones. However, I can't tackle this on my own.

The generator needs to understand a heap
of highly  platform specific details, and a thus a lot
of people need to try it on many platforms to provide
the feedback needed to make it configurable enough to
actually work on many platforms.

Any comments appreciated.


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Richard Jones]

John,

This is of course brilliant news if it can be made to work.  I can
help you with testing - I've got a few different platforms around
here.

I'm interested to know, though: how do you deal with ownership of
objects: eg. The struct returned from gethostbyname(3) is static and
so shouldn't be managed/freed from OCaml; whereas getnameinfo(3)
requires you to pass in allocated buffers and their sizes.  This
information isn't available in the header files ...

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
MAKE+ is a sane replacement for GNU autoconf/automake. One script compiles,
RPMs, pkgs etc. Linux, BSD, Solaris. http://www.annexia.org/freeware/makeplus/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Basile Starynkevitch local]

On Tue, May 18, 2004 at 06:38:37PM +1000, skaller wrote:
> This is a brief status report concerning the Felix
> automatic wrapper generator (reason for posting to Caml list
> explained below).
> 
> With some platform specific hackery, I am now able to 
> wrap 90% of all headers in '/usr/include' and execute
> a single Felix test case:
> 
> -----------------------
> include "stdio_h";
> C_hack::ignore(stdio_h::printf(c"%s\n", c"Hello World"));
> --------------------
> 
> Line 1 includes the wrapper generated for stdio.h.
> Line 2 invokes the wrapper for a popular function 'printf()'.
> This program compiles and executes correctly. [....]
> 
> It isn't clear I have got everything right yet,
> there is still a long way to go. However I do consider
> this a reasonable "proof-of-principle" demonstration
> that automatic wrapping of the complete C development
> environment is possible. The wrapper generator uses the
> frontc/Cil parser, and generates the wrappers for 
> of /usr/include in a couple of seconds.
> 
> RELEVANCE TO OCAML.
> ------------------
> 
> There isn't much that I'm doing for Felix wrappers
> that can't be applied to Ocaml: perhaps with some
> extra work it seems possible to make an Ocaml back end
> to generate Foreign Function Interfaces for the whole
> C environment in a few seconds.  [...]

very good, but I suggest a more difficult test: GTK2 (or at least the
contained Glib2 library).

Are you able to parse GTK2 or Glib2 or unistd.h or Xlib.h with your
machinery, and to generate sensible glue code for these?

I tend to believe that while <stdio.h> has some intricacies (in
particular, the variadic printf & scanf routines) there are other
header files which are more complex to parse automatically.

How do you handle C pointers, and more importantly, how do you
separate the C pointers which carry a result - for instance like in
scanf("%d",&i) and those which point to a composite input argument -
for instance the fd_set pointer arguments to select? I tend to believe
that both cases should be interfaced very differently in Ocaml


Anyway, bravo for the good work

Regards.
-- 
Basile STARYNKEVITCH -- basile dot starynkevitch at inria dot fr
Project cristal.inria.fr - phone +33 1 3963 5197 - mobile 6 8501 2359
http://cristal.inria.fr/~starynke --- all opinions are only mine 

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Olivier Andrieu]

 skaller [18 May 2004]:
 > This is a brief status report concerning the Felix
 > automatic wrapper generator (reason for posting to Caml list
 > explained below).
[...]
 > Note of course the generated wrappers are,
 > at least currently, LOW LEVEL and thus not safe.
 > 
 > for some time, people have been crying for 'MORE LIBRARIES'
 > for Ocaml.
 > 
 > Here's your chance to get THE LOT in one go.

I really don't buy this. There is just not enough information in the C
prototypes to give you a reasonable interface for OCaml. Or does your
generator use annotations (like camlIDL does) ?

For instance, consider this prototype:
  int foo (double *, int);

- the double* can be an array of doubles, an out parameter, an in/out
  parameter
- the int parameter could be the length of the array, or a regular
  parameter
- the int return type could really be a boolean

Even if (double *, int) represents an array and its length, the caml
interface is not obvious :
  float array -> int -> ...   (straightforward mapping)
  float array -> ...          (use the array length for the int parameter)
  float array -> pos:int -> len:int -> ... (account for pointer arithmetic)

In the end you get at best a very low-level bunch of autogenerated
unsafe externals, that you have to re-wrap in ocaml to get something
comfortable and/or safe to program with.

-- 
   Olivier

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Fermin Reig]

On Tue, 2004-05-18 at 09:38, skaller wrote:
> This is a brief status report concerning the Felix
> automatic wrapper generator 
[...]
> Any comments appreciated.

You might want to read this paper:

No-Longer-Foreign: Teaching an ML compiler to speak C  natively
Matthias Blume 
Electronic Notes in Theoretical Computer Science 59 No. 1 (2001)
http://www.elsevier.nl/locate/entcs/volume59.html

HTH,
Fermin


This message has been scanned but we cannot guarantee that it and any
attachments are free from viruses or other damaging content: you are
advised to perform your own checks.  Email communications with the
University of Nottingham may be monitored as permitted by UK legislation.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Tue, 2004-05-18 at 18:58, Richard Jones wrote:
> John,
> 
> This is of course brilliant news if it can be made to work. 

I think it can be "made" to work. By 'made' I mean: C headers
are relatively unstructured hacks, some important header
files actually contain *syntax* errors for example!

So it is to be expected the client will need to apply some
directives to guide the wrapping, so as to 'undo' the C hackery
if you like.

>  I can
> help you with testing - I've got a few different platforms around
> here.

Great! Note: there is currently no Ocaml back end.

> I'm interested to know, though: how do you deal with ownership of
> objects: eg. The struct returned from gethostbyname(3) is static and
> so shouldn't be managed/freed from OCaml; whereas getnameinfo(3)
> requires you to pass in allocated buffers and their sizes.  This
> information isn't available in the header files ...

Simple. I don't. For a low level C interface, everything
works exactly 'as if in C'. If you need to create a string
you call 'malloc' and 'strcpy', just as in C.

So, if there is a static data structure and you call
a function that returns it, you have to manage
it from Felix or Ocaml exactly as in C: don't free it
or you'll core dump.

Note that this couldn't possibly work in Ocaml except
that the garbage collector is smart and *knows* which
pointers to leave alone. Those pointers you manage
'just as you would do in C'.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Tue, 2004-05-18 at 19:06, Basile Starynkevitch local wrote:

> very good, but I suggest a more difficult test: GTK2 (or at least the
> contained Glib2 library).
> 
> Are you able to parse GTK2 or Glib2 or unistd.h or Xlib.h with your
> machinery, and to generate sensible glue code for these?

Yes, it parses the whole of GTK2 and unistd.h and Xlib.h,
433 files total. I am not responsible for this though:
the parsing is done by frontc, and the parse tree is then
massaged by Cil. All I did here was repackage
other peoples work.

Some files don't parse correctly, and some
have clashes, a couple are serious.

Whether the resultant 'glue code' is correct 
I do not know yet. The techniques are the same
as the SWIG based generator I was using previously,
and I managed to correctly call some gdk functions
returning screen dimensions.

> How do you handle C pointers, and more importantly, how do you
> separate the C pointers which carry a result - for instance like in
> scanf("%d",&i) and those which point to a composite input argument -
> for instance the fd_set pointer arguments to select? I tend to believe
> that both cases should be interfaced very differently in Ocaml

No distinction is made. You are right that we would like to 
build higher level interfaces than this tool can currently
generate.

It is possible to apply *some* rules in a more
advanced version of the tool. These rules would handle
type mapping, ownership, and perhaps other things like
input/output distinctions for pointers.

However at some stage, the only way to make a genuinely
Ocaml centric version of a library such as Gtk is to write
the wrapper code by hand.

A low level generator is still useful though.
It can help by importing the C interface into Ocaml,
so at least you can try to do most of the remodelling
to a higher level interface in Ocaml.

And it may still be useful, especially if you just
need one or two functions from some library where
a hand written wrapper doesn't exist.

Finally, a low level wrapper generator can be upgraded
with experience to handle common issues such as
typemapping.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Tue, 2004-05-18 at 19:25, Olivier Andrieu wrote:
>  
>  > Here's your chance to get THE LOT in one go.
> 
> I really don't buy this. There is just not enough information in the C
> prototypes to give you a reasonable interface for OCaml.

You're right. It doesn't generate an interface
any more 'reasonable' than the C one it is wrapping.

>  Or does your
> generator use annotations (like camlIDL does) ?

Not at present, although that will be necessary.

> For instance, consider this prototype:
>   int foo (double *, int);

> Even if (double *, int) represents an array and its length, the caml
> interface is not obvious :
>   float array -> int -> ...   (straightforward mapping)
>   float array -> ...          (use the array length for the int parameter)
>   float array -> pos:int -> len:int -> ... (account for pointer arithmetic)

Yup. So what you actually get 'initially' is:

	c_double c_ptr -> c_int -> c_int

Not a single Ocaml native data type is visible.

[Well, we could typemap c_int to 'native_int' since they're
isomorphic by specification, and c_double to float for the same 
reason ..]

>In the end you get at best a very low-level bunch of autogenerated
> unsafe externals, that you have to re-wrap in ocaml to get something
> comfortable and/or safe to program with.

That is correct, that's what you get.

(1) It's better than nothing :)

(2) It can be used to build a better interface *in Ocaml*

(3) Once it works, it may be possible to add some annotations
to improve the level of abstraction of the generated interface


-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Tue, 2004-05-18 at 19:38, Fermin Reig wrote:
> On Tue, 2004-05-18 at 09:38, skaller wrote:
> > This is a brief status report concerning the Felix
> > automatic wrapper generator 
> [...]
> > Any comments appreciated.
> 
> You might want to read this paper:
> 
> No-Longer-Foreign: Teaching an ML compiler to speak C  natively
> Matthias Blume 
> Electronic Notes in Theoretical Computer Science 59 No. 1 (2001)
> http://www.elsevier.nl/locate/entcs/volume59.html

unfortunately that URL doesn't work for me, I get sent to 
the main search page, and can't find anything
by Matthias Blume.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Felix Winkelmann]

skaller wrote:

> On Tue, 2004-05-18 at 19:38, Fermin Reig wrote:
> 
>>On Tue, 2004-05-18 at 09:38, skaller wrote:
>>
>>>This is a brief status report concerning the Felix
>>>automatic wrapper generator 
>>
>>[...]
>>
>>>Any comments appreciated.
>>
>>You might want to read this paper:
>>
>>No-Longer-Foreign: Teaching an ML compiler to speak C  natively
>>Matthias Blume 
>>Electronic Notes in Theoretical Computer Science 59 No. 1 (2001)
>>http://www.elsevier.nl/locate/entcs/volume59.html
> 
> 
> unfortunately that URL doesn't work for me, I get sent to 
> the main search page, and can't find anything
> by Matthias Blume.
> 

http://people.cs.uchicago.edu/~blume/papers/nlffi.pdf

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[John Chu]

Google turns up this link for No-Longer-Foreign: Teaching an ML 
compiler to speak C  natively by Matthias Blume:

http://people.cs.uchicago.edu/~blume/papers/nlffi.pdf

					john

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Tue, 2004-05-18 at 20:58, John Chu wrote:
> Google turns up this link for No-Longer-Foreign: Teaching an ML 
> compiler to speak C  natively by Matthias Blume:
> 
> http://people.cs.uchicago.edu/~blume/papers/nlffi.pdf

Thx to all those who, being smarter than I am,
tried google instead of the site's own search engine :)
I have a copy now!

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Richard Jones]

On Tue, May 18, 2004 at 08:25:41PM +1000, skaller wrote:
> Yes, it parses the whole of GTK2 and unistd.h and Xlib.h,
> 433 files total. I am not responsible for this though:
> the parsing is done by frontc, and the parse tree is then
> massaged by Cil. All I did here was repackage
> other peoples work.
> 
> Some files don't parse correctly, and some
> have clashes, a couple are serious.
> 
> Whether the resultant 'glue code' is correct 
> I do not know yet. The techniques are the same
> as the SWIG based generator I was using previously,
> and I managed to correctly call some gdk functions
> returning screen dimensions.

The actual problem with Gtk revolves (yet again) around memory
management.  Gtk uses a rather half-baked reference counting scheme.
The problems with this are similar to the problems with interfacing
Perl (see below).

> However at some stage, the only way to make a genuinely
> Ocaml centric version of a library such as Gtk is to write
> the wrapper code by hand.
> 
> A low level generator is still useful though.
> It can help by importing the C interface into Ocaml,
> so at least you can try to do most of the remodelling
> to a higher level interface in Ocaml.

It sounds very similar to Perl4caml, which offers two levels of
interface.  At the lowest level you get to manipulate SVs, HVs and AVs
directly from OCaml [1].  You have to convert strings/ints to SVs when
calling Perl code (using sv_of_string, etc.), and when returning
values you have to convert them from SVs to OCaml native types (using
string_of_sv, etc.).  For some libraries I've written high-level
wrappers which do all this conversion for you, allowing you to use
objects and modules which look just like OCaml native objects [2].

Perl uses reference counting.  I still haven't worked out a scheme to
make reference counting behave well with the OCaml garbage collector,
so the current version of Perl4caml will never deallocate Perl objects
(there is an experimental makefile flag to turn deallocation on, but
this sometimes causes programs to crash).  I've tried wrapping the
Perl objects in custom blocks, but it doesn't work, and I'd appreciate
some help sorting it out!

Rich.

[1] The low-level interface:
http://www.merjis.com/developers/perl4caml/html/Perl.html

[2] Example of a high-level wrapper:
http://www.merjis.com/developers/perl4caml/html/Pl_HTML_Element.html
http://www.merjis.com/developers/perl4caml/html/Pl_HTML_Element.html_element.html

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://www.winwinsales.co.uk/ - CRM improvement consultancy

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Michael Hamburg]

>
> It sounds very similar to Perl4caml, which offers two levels of
> interface.  At the lowest level you get to manipulate SVs, HVs and AVs
> directly from OCaml [1].  You have to convert strings/ints to SVs when
> calling Perl code (using sv_of_string, etc.), and when returning
> values you have to convert them from SVs to OCaml native types (using
> string_of_sv, etc.).  For some libraries I've written high-level
> wrappers which do all this conversion for you, allowing you to use
> objects and modules which look just like OCaml native objects [2].
>
> Perl uses reference counting.  I still haven't worked out a scheme to
> make reference counting behave well with the OCaml garbage collector,
> so the current version of Perl4caml will never deallocate Perl objects
> (there is an experimental makefile flag to turn deallocation on, but
> this sometimes causes programs to crash).  I've tried wrapping the
> Perl objects in custom blocks, but it doesn't work, and I'd appreciate
> some help sorting it out!

I don't know how Perl's reference-counting scheme works, but I'm  
working on interfacing O'Caml with Cocoa, which also uses ref-counts.   
I've basically made O'Caml incref any object allocated on its heap, and  
decref it as a finalization step.  Conversely, O'Caml objects in C get  
added to the global roots.  I still have to do more testing, but it  
seems that this doesn't crash, nor does it leak much memory.

I say much because such a scheme cannot detect loops.  I think you'd  
have to mod the garbage collector to do so.  As of now, there's also no  
way to safely break a known loop, since decrefing an object may cause  
it to go away before O'Caml can finalize it, which would cause a  
segfault.  An interface change to the  Cocoa wrapper of O'Caml objects  
might fix this.

Mike

> Rich.
>
> [1] The low-level interface:
> http://www.merjis.com/developers/perl4caml/html/Perl.html
>
> [2] Example of a high-level wrapper:
> http://www.merjis.com/developers/perl4caml/html/Pl_HTML_Element.html
> http://www.merjis.com/developers/perl4caml/html/ 
> Pl_HTML_Element.html_element.html
>
> -- 
> Richard Jones. http://www.annexia.org/ http://www.j-london.com/
> Merjis Ltd. http://www.merjis.com/ - improving website return on  
> investment
> http://www.winwinsales.co.uk/ - CRM improvement consultancy

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Wed, 2004-05-19 at 03:21, Michael Hamburg wrote:
> >
> > It sounds very similar to Perl4caml, 

> I don't know how Perl's reference-counting scheme works, but I'm  
> working on interfacing O'Caml with Cocoa, 

Both these problems are more difficult than interfacing
Ocaml with C, if only because there are THREE languages
involved, and C is only the glue in between. So there
is a requirement to translate object models and memory
models.

A low level C/Ocaml interface is simpler by simply
specifying NO translations or memory model changes.

It's typical to desire stub functions that map
data types across boundaries. The approach I'm using
is to simply NOT do this at all.

Instead, you can write the type conversions as
auxilliary functions in C, which can be called
from Ocaml:

	let s = c_str "Ocaml string" in
	let d = c_malloc (200) in
	c_strcpy(d,s);

Note that the conversion from an Ocaml string to a 
C string (null terminated char pointer thingo) is done
by a separate function, this has nothing to do with
the wrapped up functions c_strcpy and c_malloc.

Any memory management issues that arise from using
c_str, c_malloc, and c_strcpy are the clients.
The FFI makes no attempt to do any memory management,
it makes no attempt to do any type conversions.

What makes this 'a good thing' is precisely how
totally brain dead it is. You know excactly what it will
do -- the same thing the underlying C does (modulo
a need for boxing).

I think the boxing is transparent as follows:
When you need to pass a C struct to a function --
and I mean *pass by value* -- you actually have
a Caml pointer to a custom block. C code casts
the custom block to form the argument .. the C calling
convention will copy it onto the stack.

At this point, if the custom block becomes unreachable
it gets collected, but that does not matter: C has a
copy on the stack. Returning a value requires copying
the struct off the stack into a custom block,
and returning a pointer instead, again, managed by Ocaml.

Pointers themselves are values .. so the above applies
to them too the same way. It should all work just fine
because C *only* allows pass by value: the C *never*
sees anything actually inside a custom block, it only
ever sees a copy.

There is no nesting. When you pass a pointer to an array,
the pointer is boxed. Ocaml owns the box, the pointer is copied
onto the stack and is owned by C, and the array is owned
by whoever created it .. typically C, and probably malloc.
The array itself doesn't live in a custom block so it isn't
managed by Ocaml, rather it is managed by C + client Ocaml code,
the latter by calling C .. 

As far as I can see this has to work in all cases,
it simply cannot fail and the wrapper can be generated
completely automatically. Of course the client Ocaml code
can 'free' a null pointer just the same can be done
with the equivalent C code: the assurance the wrapper
generator works comes from a simple isomorphism which 
also ensures all the faults of the C memory management
methods are also reflected.

The big plus here is that you can simply use any C library
from Ocaml 'as if writing C' so there is nothing to learn
about a complex new safe interface.

I expect *some* typemapping might be possible: certainly
where the object models coincide it may be possible
to skip the abstraction and just pass the common native
representation around, but I think that's limited to
some numeric cases plus some system types (like unix
file handles perhaps?)

BTW: Felix doesn't generally use boxing, so it will be 
at least one level of indirection faster than Ocaml.
It might be more though, since 'inlining' is still possible
with Felix, since it generates source and the underlying
C compiler can still do it's own optimisations.
So for Felix, the term 'Foreign Function Interface' is slightly
misleading, since C functions aren't foreign, and the wrapping
is primarily syntactic sugar including type annotations,
rather than requiring anything executable. The principal cost
is the loss of automatic intensional polymorphism
(since that requires boxing .. )

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Richard Jones]

I think your analysis is correct, except that you've missed out C
macros.  Both the Perl and Gtk C interfaces use macros all over the
place as a kind of inline function.

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
http://www.winwinsales.co.uk/ - CRM improvement consultancy

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Gerd Stolpmann]

On Die, 2004-05-18 at 14:11, Richard Jones wrote:
> The actual problem with Gtk revolves (yet again) around memory
> management.  Gtk uses a rather half-baked reference counting scheme.
> The problems with this are similar to the problems with interfacing
> Perl (see below).
> 
> > However at some stage, the only way to make a genuinely
> > Ocaml centric version of a library such as Gtk is to write
> > the wrapper code by hand.
> > 
> > A low level generator is still useful though.
> > It can help by importing the C interface into Ocaml,
> > so at least you can try to do most of the remodelling
> > to a higher level interface in Ocaml.
> 
> It sounds very similar to Perl4caml, which offers two levels of
> interface.  At the lowest level you get to manipulate SVs, HVs and AVs
> directly from OCaml [1].  You have to convert strings/ints to SVs when
> calling Perl code (using sv_of_string, etc.), and when returning
> values you have to convert them from SVs to OCaml native types (using
> string_of_sv, etc.).  For some libraries I've written high-level
> wrappers which do all this conversion for you, allowing you to use
> objects and modules which look just like OCaml native objects [2].
> 
> Perl uses reference counting.  I still haven't worked out a scheme to
> make reference counting behave well with the OCaml garbage collector,
> so the current version of Perl4caml will never deallocate Perl objects
> (there is an experimental makefile flag to turn deallocation on, but
> this sometimes causes programs to crash).  I've tried wrapping the
> Perl objects in custom blocks, but it doesn't work, and I'd appreciate
> some help sorting it out!

Maybe a look at the WDialog interface for Perl helps:

http://cvs.sourceforge.net/viewcvs.py/wdialog/wdialog/code/src/UI/Stubs/stubs.ml?rev=3.4.2.1&only_with_tag=wdialog-perlapi-030403-branch&view=auto

(Note: This is only in the wdialog-perlapi-030403-branch, and not in the
released version of WDialog.)

This interface exports O'Caml objects as Perl objects, and not the other
way round like Perl4caml. The concept is:

- The objects exist when they exist on the O'Caml side. The Perl 
  object is only a proxy for the O'Caml object.

  The existence is explicitly controlled by the global variable
  uiobjects_list; all existing objects are added to it.

- The O'Caml object has a reference to the Perl proxy, but the Perl
  proxy knows only the ID of the O'Caml object that can be used to
  look it up in uiobjects_list. This is important: In one direction
  one can use addresses to reference, but in the other direction
  one must use a weak name like object IDs.

- There is a finaliser for the O'Caml object. When it is called, the
  reference counter of the Perl proxy is decreased. Usually, the
  counter goes to zero then, and the Perl proxy will be deallocated.
  It may happen, however, that the proxy lives longer than the
  object. However, when one calls methods of such half-dead proxies,
  exceptions are raised, so this is not a problem. A half-dead
  proxy can be detected because the object ID cannot be looked up.

- The remaining question is when an object should be removed from
  uiobject_list. Fortunately, there is a kind of activity cycle, and
  when the cycle is over, it is possible to set uiobject_list:=[].

This solution may be interesting for you, because it demonstrates some
useful techniques like finalisers, and because one can see which
questions must be answered. Btw, the code is fully debugged, and passes
a test showing that all memory is actually freed after the activity
cycle.

I think the point is that this API selects one of the memory managers as
the primary one. For a number of reasons the O'Caml manager was chosen.
In the other language one has only proxies that route accesses across
the language boundary.

A major problem are cyclic references between the managers. When they
cannot be avoided, e.g. to realise callbacks across the language
boundary, the primary manager must record these references explicitly,
and give them a weak name (like the object IDs). It may happen that the
weak name cannot be resolved, and this is handled as a runtime error.
The weak names must be allocated and deallocated explicitly by user code
(Wdialog binds this to the activity cycle, but such a thing is not
available in the general case.)

I hope this gives you some idea how to tackle this problem.

Gerd
-- 
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
------------------------------------------------------------

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

[Caml-list] Functional critical section

[SWAMPY]

Hello!

   I'm very new to ocaml, and I'm trying to figure out how
to do something that may or may not actually be possible.
Any suggestions?

    I'd like to write

        <name> <expr>

    as an expression, and have this evaluate <expr> first
evaluating fixed expression A, and then after evaluting
<expr> evaluate fixed expression B.
    Then, I could do something like:

    critical (threadsafeFunction arg1 arg2 arg3)

    and have this do the equivalent of

    Mutex.lock someMutex; let tmp = threadsafeFunction arg1
arg2 arg3; Mutex.unlock someMutex; tmp;;

    Is this sort of thing possible?  In general I'm
interested in techniques I could use to wrap
imperative-biased parts of the library to more functional
constructs.

    Cheers,

    David J. Trombley

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Functional critical section

[Kenneth Knowles]

With Camlp4 you may be able to do this via pre-processing.  Otherwise you could
do it with lazy evaluation, supported by the Lazy module in the standard
library.  An inefficient lazy evaluation can be had be just adding an additional
unit argument to your function.  (lablGL does something like this, though I'm
not sure of their motivations)

http://caml.inria.fr/camlp4/manual/
http://wwwtcs.inf.tu-dresden.de/~tews/htmlman-3.07/libref/Lazy.html

Kenn

On Tue, May 18, 2004 at 04:10:44PM -0400, SWAMPY wrote:
> Hello!
> 
>    I'm very new to ocaml, and I'm trying to figure out how
> to do something that may or may not actually be possible.
> Any suggestions?
> 
>     I'd like to write
> 
>         <name> <expr>
> 
>     as an expression, and have this evaluate <expr> first
> evaluating fixed expression A, and then after evaluting
> <expr> evaluate fixed expression B.
>     Then, I could do something like:
> 
>     critical (threadsafeFunction arg1 arg2 arg3)
> 
>     and have this do the equivalent of
> 
>     Mutex.lock someMutex; let tmp = threadsafeFunction arg1
> arg2 arg3; Mutex.unlock someMutex; tmp;;
> 
>     Is this sort of thing possible?  In general I'm
> interested in techniques I could use to wrap
> imperative-biased parts of the library to more functional
> constructs.
> 
>     Cheers,
> 
>     David J. Trombley
> 
> -------------------
> To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
> Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Functional critical section

[Evan Martin]

On Tue, May 18, 2004 at 04:10:44PM -0400, SWAMPY wrote:
>    I'm very new to ocaml, and I'm trying to figure out how
> to do something that may or may not actually be possible.
> Any suggestions?
> 
>     I'd like to write
> 
>         <name> <expr>
> 
>     as an expression, and have this evaluate <expr> first
> evaluating fixed expression A, and then after evaluting
> <expr> evaluate fixed expression B.

I'm not sure what you mean by A and B here, but in general there is a
specific way* things are evaluated and the only way to make things happen
another way is to use closures to delay evaluation.

>     Then, I could do something like:
> 
>     critical (threadsafeFunction arg1 arg2 arg3)
> 
>     and have this do the equivalent of
> 
>     Mutex.lock someMutex; let tmp = threadsafeFunction arg1
> arg2 arg3; Mutex.unlock someMutex; tmp;;
> 
>     Is this sort of thing possible?  In general I'm
> interested in techniques I could use to wrap
> imperative-biased parts of the library to more functional
> constructs.

Sure, that's the sort of thing that anonymous functions are great for.

let criticial f =
  Mutex.lock someMutex;
  let tmp = f () in
  Mutex.unlock someMutex;
  tmp

And then call it like this:

 criticial (fun () -> threadsafeFunction arg1 arg2 arg3)

The "trick" here is that we wrap up the inner in its own function, so
that the function "critical" can decide when to run it-- between the
mutex lock and unlock.



* A more pedantic description of evalution order, as I understand it:
Arguments are always evaluated before calling functions.
However, in the application form "e1 e2", the thing being applied (e1)
must also be evaluated to a value, and the order of those evaluations is
either undefined or right to left (I forget which; maybe the former
officially, and the latter in practice?).

That means that in something like e1 e2, e1 and e2 are evaluated to
values, and then the function that e1 evaluates to is evaluated with
e2's value as its argument.

In most situations, e1 is just an identifier like "threadsafeFunction",
and evaluation of it is trivial.  So it doesn't matter whether e2
evaluates before or after that, and the actual application of the
function is always last.

If you ever want to force evaluation order, you can use let.
For example, if I have code like:
 f (function1 1) (function2 31)
and I want to be certain that function2 is called first, you can rewrite
it to
 let f2 = function2 31 in
 f (function1 1) f2

-- 
Evan Martin
martine@danga.com
http://neugierig.org

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Wed, 2004-05-19 at 05:27, Richard Jones wrote:
> I think your analysis is correct, except that you've missed out C
> macros.  Both the Perl and Gtk C interfaces use macros all over the
> place as a kind of inline function.

Flxcc currently forgets all about macros.

I'm not totally unhappy :)

You may have to call __f instead of f.
That's not quite as bad as losing M_PI out of math.h though.

To fix this I'd need a C preprocessor to modify,
and probably also some annotations to help chose
the macros to export as part of the interface.

They'd have to be typed.. which isn't so easy if the macros
are used as generics.

Felix itself has no problem with binding macros,
including generic ones (just cheat a bit and pretend
they're really parametrically polymorphic).

Note I haven't forgotten them. Indeed one of my main
reasons for abandoning SWIG was that it doesn't
provide any hooks to catch macros intended as
functions. (It catches constants automatically though).

The bottom line is that if you want to make a wrapper
that conforms to a well specified (documented) interface,
you should do it 'by hand' or by processing the documentation,
not by processing the source code. Of course probably
the fastest way to do this is to patch up a flxcc generated
wrapper.. :)

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Functional critical section

[thornber]

On Tue, May 18, 2004 at 04:10:44PM -0400, SWAMPY wrote:
>     Then, I could do something like:
> 
>     critical (threadsafeFunction arg1 arg2 arg3)
> 
>     and have this do the equivalent of
> 
>     Mutex.lock someMutex; let tmp = threadsafeFunction arg1
> arg2 arg3; Mutex.unlock someMutex; tmp;;

I use a 'with_resource' function:

| let try_finally fn fin =
|   try
|     let result = fn () in
|     fin ();
|     result;
|   with x -> fin (); raise x
| 
| let with_resource resource fn post =
|   try_finally (fun () -> fn resource) (fun () -> post resource)


Which could be used to make a 'with_file' or 'with_mutex' fn:

| let with_file file fn =
|   with_resource (open_in file) fn close_in
|
| let with_mutex fn =
|   with_resource (Mutex.lock someMutex) fn Mutex.unlock

- Joe

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Functional critical section

[thornber]

On Tue, May 18, 2004 at 01:39:44PM -0700, Evan Martin wrote:
> Sure, that's the sort of thing that anonymous functions are great for.
> 
> let criticial f =
>   Mutex.lock someMutex;
>   let tmp = f () in
>   Mutex.unlock someMutex;
>   tmp

If f throws an exception the mutex will not get unlocked, which is
probably not what you wanted.

- Joe

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Marcin 'Qrczak' Kowalczyk]

W liście z wto, 18-05-2004, godz. 18:38 +1000, skaller napisał:

> With some platform specific hackery, I am now able to 
> wrap 90% of all headers in '/usr/include' and execute
> a single Felix test case:

It's impressive, but I'm afraid a high quality binding to an average
library is impossible to obtain automatically.

I'm currently wrapping some libraries for my own language. I have
working bindings to zlib, ncurses, Python interpreter, some Unix API,
and as a rather impractical but instructive example - qsort().

It's a tedious task because I set up high requirements for myself.
Big ints are transparently converted even in case the C library uses
64-bit numbers, memory management is integrated with GC, various error
codes are converted to exceptions, exceptions are propagated through C
callbacks, C enumerations and ints which encode enumerations are
translated in a typesafe manner (my language is dynamically typed,
so it looks differently than in OCaml, but the issue is there too).

More importantly, C libraries which can be made instances of generic
APIs of my language are wrapped appropriately: you can use zlib as a
layer on I/O streams, or treat gzip files with the same API as native
files, use Python collections and numbers together with collections and
numbers of my language, mixed in either environment, and use values
hashed by one language as keys in dictionaries in the other language.

All this seems impossible to be done automatically, and I feel it's
important.

I wonder how much the amount of work depends on the source and target
languages. For example although interfacing to the Python interpreter,
written in C, was a big task, interfacing to particular Python libraries
themselves need zero additional code! And they even look quite usable,
modulo lack of matching of additional generic interfaces. I have a
working Gtk+ binding for free because Python people once wrapped it for
Python.

OCaml doesn't seem to use generic APIs with substitutable interfaces too
much, because it's expressed either as modules, which are somewhat
second-class, or as objects, which are quite heavy. For example there is
no place to offer your semantics of arithmetic, to provide a layer of
translation for I/O functions, or a type of a sequence. This is both a
good and a bad thing:
- Poor reuse of code, you can't use standard function on non-standard
  objects, e.g. character I/O works only on raw files.
+ No runtime dispatch, so it's fast.
+ Less work for interface generator :-)

I will try to make a binding between my language and OCaml, but I'm
not sure how should I approach it because OCaml is statically typed.
Perhaps this time I should consider OCaml the "master" side where most
glue code is put in. Objects of a dynamically typed language are easier
to be operated on from other languages.

It will be fun to use Python libraries from OCaml through my language.
I've seen some Python binding for OCaml but it was very poor.

-- 
   __("<         Marcin Kowalczyk
   \__/       qrczak@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Sat, 2004-05-22 at 20:09, Marcin 'Qrczak' Kowalczyk wrote:
> W liście z wto, 18-05-2004, godz. 18:38 +1000, skaller napisał:
> 
> > With some platform specific hackery, I am now able to 
> > wrap 90% of all headers in '/usr/include' and execute
> > a single Felix test case:
> 
> It's impressive, but I'm afraid a high quality binding to an average
> library is impossible to obtain automatically.

What you mean is a high *level* binding.

The automatically generated one is the very highest quality.
It comes with a guarrantee your hand written one will never have.
It is guarranteed to work. Exactly as the underlying C does.
[And also you can construct it in a few seconds :]

> All this seems impossible to be done automatically, and I feel it's
> important.

Yes. But there are many possible high level bindings.
This code generator quite deliberately, and by specification,
produces a low level binding. Its aim is to guarrantee
an isomorphism between the target language and the C semantics.

Yes, it is possible to automate a higher level binding.
But it seems wise to get the lowest possible level binding
to work first: and to make absolutely sure it can always
be made available as a fallback.

There is a very important theoretical reasoning behind this:
if you try to interface systems with disparate object models
your bindings are GUARRANTEED TO FAIL.

The reason is clear and contained directly in the assumption
that the object models are disparate :)

SWIG can't even get C++ and Python strings to work.
It never will. It isn't a bug in SWIG either.
It simply cannot be done. [Python strings are immutable,
C++ strings are not .. this is enough to guarrantee
no isomorphism can exist]

So beware .. almost every high level binding is certain
to be flawed.

> I wonder how much the amount of work depends on the source and target
> languages. For example although interfacing to the Python interpreter,
> written in C, was a big task, 

Felix binding to Python will work out of the box,
meaning the automatically generated one will work.

It won't convert Felix strings to Python ones inside
the FFI. It won't touch any ref counts. It will do EXACTLY
what calling the C API would do.

The big win here is that you can now write your high level
interface in the target language (with a library of 
specially written C utilities to help).

BTW: to give you an idea of 'low level':
the code to drive, for example, GTK, in Felix will
actually be MORE COMPLEX than the C.

The reason is Felix doesn't allow any automatic conversions,
whilst C does. You have to add casts physically where you
need them.

So this is *really* low level. Deliberately.
I don't want to write high level bindings in C.
I'd rather do it all in Felix. And ditto for an Ocaml
binding: I'd rather do the high level binding logic
in Ocaml than try to do it in some arbitrary mix
of the two, where a heap of heavy design work is needed
to get the boundary just right. Much easier to modify
code written in a single language.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Marcin 'Qrczak' Kowalczyk]

W liście z sob, 22-05-2004, godz. 23:13 +1000, skaller napisał:

> It comes with a guarrantee your hand written one will never have.
> It is guarranteed to work. Exactly as the underlying C does.

The binding itself - indeed, but code using it is not guaranteed
to work :-)
It's more easy to make errors when using a low-level binding.

> Yes, it is possible to automate a higher level binding.

My point is that it's hard to automate. But it can be done by hand.

There might be tools which help a bit (like SWIG; I haven't used it).
They will never produce a high-level binding without a separate
interface description and some amount of hand-written code.

> There is a very important theoretical reasoning behind this:
> if you try to interface systems with disparate object models
> your bindings are GUARRANTEED TO FAIL.

Not quite. There are three ways:

1. Your automatic low-level binding. It is an interface between
   different object models, and it works, although it might not be
   most convenient to use.

2. My hand-written high-level binding. It's designed separately for
   different libraries, and it is often possible to make most of the
   library functionality available in the target language under proper
   conventions. This again contradicts your claim; the point is that
   it doesn't have to be universal, working for all libraries.

3. A cross between the two: a hypothetical universal translator which
   is written once and produces bindings for arbitrary libraries,
   like yours, but the code integrates with conventions of the target
   language, like mine. This is indeed doomed to fail. But with some
   languages you can come much closer than with C (e.g. with Python,
   as I did).

> SWIG can't even get C++ and Python strings to work.
> It never will. It isn't a bug in SWIG either.
> It simply cannot be done. [Python strings are immutable,
> C++ strings are not .. this is enough to guarrantee
> no isomorphism can exist]

So it must be designed case by case:

- In some C++ libraries the mutability of a given string parameter or
  result doesn't matter, and it can be converted by value.

- In others it does matter, and a C++ string should be wrapped in a
  Python object.

Surely you don't claim that it's impossible to use a C++ library using
strings in Python, or vice versa!

> The big win here is that you can now write your high level
> interface in the target language (with a library of 
> specially written C utilities to help).

Unfortunately in my case it has two problems:

1. My compiler doesn't produce as optimized code as it could. It's not
   that bad, but dynamic typing make generating good code very hard.

2. Dynamic typing also implies that the binding would be very unsafe.
   My binding to almost all libraries is safe, because it performs the
   necessary checks, but a low-level universal binding would be not.
   Static types can make it somewhat safer.

3. Since some C functionality, like calling functions of varying
   types, requires compiling new code (or using non-portable hacks),
   the binding would either have to use macros (which I haven't
   implemented yet) or be implemented inside the compiler (which
   requires some work too, perhaps much).

It would be interesting to try to make some universal C binding for my
language (after macros). I'm leaving considering it for the future.

-- 
   __("<         Marcin Kowalczyk
   \__/       qrczak@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Sun, 2004-05-23 at 00:19, Marcin 'Qrczak' Kowalczyk wrote:


> Unfortunately in my case it has two problems:
> 
> 1. My compiler doesn't produce as optimized code as it could. It's not
>    that bad, but dynamic typing make generating good code very hard.

> It would be interesting to try to make some universal C binding for my
> language (after macros). I'm leaving considering it for the future.

What I think you will have to do is what I'm doing now
with Felix. YOu actually commit to writing the bindings
in your language. You make sure you can make them work.

Then, you examine what would make it possible
for the compiler to generate better code and extend
your language to handle it.

For example, Felix has a specialised 
'domain specific sublanguage' which deals with C bindings.
There is a whole class of statements which do nothing
else than organise C interfacing.

In your case, you might consiser a DSSL which has
some static typing, just to deal with FFIs ..
but which is higher level than C, and interfaces
with the rest of your language.

I think you can do this by saying 'we have many objects
at the moment, all the same static type -- my language
isn't typeless, just monotyped'.

Ok, so add a type system with two types: native and foreign.
You can't lose any dynamics here, since they're embodied
in the native part already. Just a thought. I worked
on statically typing Python .. and it turned out to be
almost impossible but *only* because of a few nasty
language features 99% of all Python programs could do without
(unrestricted 'eval' and mutable modules being two such evils).

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[Marcin 'Qrczak' Kowalczyk]

W liście z nie, 23-05-2004, godz. 02:14 +1000, skaller napisał:

> For example, Felix has a specialised 
> 'domain specific sublanguage' which deals with C bindings.
> There is a whole class of statements which do nothing
> else than organise C interfacing.

In some sense I am doing this right now. The constructs are very
low-level though: they insert C snippets with specified names #defined
to locations of given objects, provide appropriate #defines needed to
throw exceptions from the embedded code. There are constructs for
inserting toplevel C definitions and for implementing new types.

They rely on the fact that code is compiled into C, and thus the
compiler can insert these snippets in appropriate places and compile
the resulting C code as a whole. Contents of the snippets rely on the
implementation of the runtime, and use various functions and macros
which would otherwise be internal to the runtime. For now there is a
single implementation of my language anyway, but these constructs will
not be portable. I always plan which parts of my language are meant to
be implementation-dependent, and unfortunately it includes the current
state of the FFI.

Your binding language lies between my low-level binding, and my
hand-written high-level bindings for particular libraries. We compared
your language to my high-level interfaces, but perhaps it should be
compared to my low-level sublanguage. Now mine is as universal as
yours :-) but it requires some knowledge about internals of the
compiler, comparable to what is required for OCaml extensions.

I think yours is easier to use because your runtime is much closer to
C++ than mine. I don't know the details of yours, but major differences
between my runtime and C are:
- dynamic typing
- variables refer to objects, not contain them
- garbage collection (copying, generational; pointers may change,
  pointer stores require write barrier)
- tail calls
- exceptions (as lightweight as in OCaml)
- integers are unbounded (with a separate representation for fixnums,
  but is should be transparent to the user)
- strings are Unicode (UTF-32 or ISO-8859-1 internally).

Providing a nice interface is closing a big semantic gap. I just did
bzip2 yesterday, 700 lines (streams & files).

> In your case, you might consiser a DSSL which has
> some static typing, just to deal with FFIs ..
> but which is higher level than C, and interfaces
> with the rest of your language.

I'm not sure how it should look like.

I feel it should wait for macros, because doing everything with
functions is inefficient, and implementing everything inside the
compiler is tedious.

> Ok, so add a type system with two types: native and foreign.
> You can't lose any dynamics here, since they're embodied
> in the native part already.

Hmm, interesting. I see many problems though. A native object has a
uniform representation, but foreign objects have varying sizes. What if
I apply a function to a foreign object? It seems I need also function
types.

How should a function with foreign parameters be represented? The
calling convention of native functions doesn't allow passing non-native
arguments! It passes them in virtual registers (global C variables)
because of tail calls and garbage collection.

Should a variable holding a foreign type be mutable? Should the foreign
object be copied when passing it around? Implicing copying + mutability
= weird semantics, and inconsistent with the rest of the language.

Where should a foreign object be allocated? Currently I do it manually
and have to choose: either wrap it inside a native object (possibly with
a finalizer), or - if its address must be stable - allocate it with
malloc and wrap a pointer (the finalizer may do something besides
calling free()). Providing all the flexibility in a specialized language
requires a complex language. Currently I just implement it by hand.

Sometimes I wrap some other fields together with the object. For example
in gzip & bzip2 I add two boolean fields: whether a stream has been
completely closed (and should no longer be used nor even finalized)
and whether decompression has seen end of stream (and I should not try
flushing the remaining part when closing, bzip2 doesn't like that).
It's a minor detail, but shows how hard is to do wrapping automatically.

-- 
   __("<         Marcin Kowalczyk
   \__/       qrczak@knm.org.pl
    ^^     http://qrnik.knm.org.pl/~qrczak/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Automatic wrapper generator

[skaller]

On Sun, 2004-05-23 at 20:58, Marcin 'Qrczak' Kowalczyk wrote:

> They rely on the fact that code is compiled into C, and thus the
> compiler can insert these snippets in appropriate places and compile
> the resulting C code as a whole. 

Just what Felix does :)

> Contents of the snippets rely on the
> implementation of the runtime, and use various functions and macros
> which would otherwise be internal to the runtime. For now there is a
> single implementation of my language anyway, but these constructs will
> not be portable. 

I put all the special constructs in the standard library,
in a module named 'C_hack' :-)

> Your binding language lies between my low-level binding, and my
> hand-written high-level bindings for particular libraries. We compared
> your language to my high-level interfaces, but perhaps it should be
> compared to my low-level sublanguage. 

Yes, that seems reasonable.

> Now mine is as universal as
> yours :-) but it requires some knowledge about internals of the
> compiler, comparable to what is required for OCaml extensions.

In Felix, some FFI stuff needs no knowledge of the compiler
or implementation. Most of it, in fact :)

However, you can do some nasty tricks, and do need to
sometimes, and those things are implementation dependent.
C_hack handles many of them.

> I think yours is easier to use because your runtime is much closer to
> C++ than mine. 

Yes. There are only two languages which can bind to C
better than Felix: C++ and C (in that order LOL!)

It was indeed specifically designed to be an upgrade
of C/C++. I learned a lot about compatibility on the C++
committee, particularly its *political* importance :)

> I don't know the details of yours, but major differences
> between my runtime and C are:
> - dynamic typing
> - variables refer to objects, not contain them
> - garbage collection (copying, generational; pointers may change,
>   pointer stores require write barrier)
> - tail calls
> - exceptions (as lightweight as in OCaml)
> - integers are unbounded (with a separate representation for fixnums,
>   but is should be transparent to the user)
> - strings are Unicode (UTF-32 or ISO-8859-1 internally).

Felix is a statically typed Algol-like language meaning
it has variables containing values. (Variants are
boxed due to an inexcusable flaw in C++)

It has an exact  *user supplied* garbage collector capable 
of supporting compaction (but I haven't actually written a compactor).

It optimises tail calls for procedures (but relies on g++
for functions at the moment -- this is hard to fix
because gcc is a toy compiler, and can't be relied
on to do any kind of heavy work)

There's no exception handling. 

Felix has no datatypes (not even bool).

Numbers of various kinds are provided in the library
using the binding technology .. although bool
doesn't need it:

typedef void = 0;
typedef unit = 1;
typedef bool = 2; // = 1 + 1 = unit + unit

defines it in terms of the basic type constructors.
[and by a "lucky" coincidence .. this maps to 
C/C++ int/boolean ..]

There is some hackery to make the lexer etc support
literals -- this is *supposed* to be handled
by a loadable user specified extension .. but
Ocamlyacc/lex lexers/parsers aren't so easy to
extend, and Ocaml native code is hard to extend
dynamically (Felix itself is designed from the ground
up to support dynamic loading, but bootstrapping
is a long way off yet ..)

> > In your case, you might consiser a DSSL which has
> > some static typing, just to deal with FFIs ..
> > but which is higher level than C, and interfaces
> > with the rest of your language.
> 
> I'm not sure how it should look like.

Use type inference, with a fallback to dynamic typing.
That's basically what I did for Python.

> Should a variable holding a foreign type be mutable? Should the foreign
> object be copied when passing it around? Implicing copying + mutability
> = weird semantics, and inconsistent with the rest of the language.

The simplest model for *foreign* objects is 'pointer'.
They're all the same size, and you manage them by value.

If you do some type inference, you may find your native
integers, strings, and also foreign pointers can sometimes
be detected and you can then possibly optimise things.
I guess you need to always be able to fallback on
dynamics.

> Where should a foreign object be allocated? 

You can't allocate one inside your language, its foreign ..
so it has to be done by a foreign function.

-- 
John Skaller, mailto:skaller@users.sf.net
voice: 061-2-9660-0850, 
snail: PO BOX 401 Glebe NSW 2037 Australia
Checkout the Felix programming language http://felix.sf.net



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners