[Caml-list] CamlDL/Abstract pointers problem

[Caml-list] CamlDL/Abstract pointers problem

[ronniec95]

Hi list,

I'm trying to create an FFI to a messaging interface and keep getting problems
with passing around opaque pointers from C back to Ocaml (crashes). This
is on WIN32 btw. Here's the simples example I could thing of that reproduces
the problem. Any help is appreciated!

--- test.idl ---
typedef [abstract,ptr] void* mywindow;

quote(C,"__declspec(dllexport)")
mywindow init_window();

--- end test.idl ---

--- testimpl.c --
typedef struct _Foo
{
	int x;
	char* y;
} Foo;

mywindow init_window(void)
{
	/* Initialise a structure */
	Foo* f = (Foo*) malloc(sizeof(Foo));
	f->x = 10;
	f->y = (char*)malloc(sizeof(char) * 256);
	strcpy(f->y,"hello world");
	printf("%d %s\n",f->x,f->y);
	fflush(stdout);
	return (mywindow)f; // Crashes after leaving this block
}

--- end testimpl.c ---


--- main.ml ---
let win () = Test.init_window ()
--- end main.ml ---

I'm compiling (test_stub.c (generated) and testimpl.c) with MSVC 6. with
flags : /MD /W3 /GX /O2 /D "WIN32" /D "NDEBUG" /D "_MBCS" /D "CAML_DLL"
/D "_USRDLL"

and building a cmxa (ocamlopt) file that I link with
ocamlopt -verbose -o go -I . -cclib "/link -LIBPATH:C:/Progra~1/Micros~2/vc98/lib"
test.cmxa main.cmx

which displays:
+ ml /nologo /coff /Cp /c /Fo"c:\TEMP\camlstartupca0b4c.obj" "c:\TEMP\camlstartupd503a3.asm">NUL
+ cl /nologo /MT /Fe"go" -I"e:\\Local\\ObjectiveCaml\\lib" "c:\TEMP\camlstartupca0b4c.obj"
"e:\\Local\\ObjectiveCaml\\lib\std_exit.obj" "main.obj" "test.lib" "e
:\\Local\\ObjectiveCaml\\lib\stdlib.lib" "libtest.lib" "/link" "-LIBPATH:C:/Progra~1/Micros~2/vc98/lib"
"e:\\Local\\ObjectiveCaml\\lib\libasmrun.lib" advapi32.lib


I'm thinking that either I've forgotten to initialise something or linked
with wrong options but I can't figure out what that would be. Any sample
program/makefile that works under win32 or hints would be appreciated. BTW.
Generating my own handcoded stubs appears to  work fine (though is extremely
dull).

All I want to do is just give Ocaml a pointer to something I've created
in C and pass it back to other C functions later; don't want it to do anything
with it (including moving it around the ocaml heap).\

Thanks for any help.

Ronnie

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Richard Jones]

On Thu, Jan 29, 2004 at 01:06:39PM +0000, ronniec95@lineone.net wrote:
> All I want to do is just give Ocaml a pointer to something I've created
> in C and pass it back to other C functions later; don't want it to do anything
> with it (including moving it around the ocaml heap).\

It's my understanding that you can't just pass a pointer back to OCaml
and cast it to a value.  OCaml will think that the pointer points to
an OCaml heap object, and hence try to examine / move / delete it in
the garbage collector.  Instead you need to do one of two things:
either set the bottom bit to 1, in which case OCaml will treat it as a
plain integer, or else wrap it up in a box, as in the code below.  The
Abstract_tag tells the garbage collector not to look inside the box.

static value
Val_voidptr (void *ptr)
{
  CAMLparam0 ();
  CAMLlocal1 (rv);
  rv = alloc (1, Abstract_tag);
  Field(rv, 0) = (value) ptr;
  CAMLreturn (rv);
}

#define Voidptr_val(type,rv) ((type *) Field ((rv), 0))

You need to call Val_voidptr to wrap up your C void* pointer into an
OCaml box, and Voidptr_val to unwrap and get your pointer back again.

In any case you need to also add lots of calls to Gc.full_major () to
your code, which will trigger an early core dump if there is some
memory corruption.

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
If I have not seen as far as others, it is because I have been
standing in the footprints of giants.  -- from Usenet

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[ronniec95]

I understand that bit ok. But what I wanted to know is why CAMLIDL was not
doing this for me... as I said in my original email, I'm quite happy handwriting
all this stuff (Abstract_tag/Custom_tag all ok for me )- it works fine,
but is quite dull!

I specifically wanted to know what it is about camlidl which stops it generating
the correct code for this.

Ronnie

>-- Original Message --
>Date: Thu, 29 Jan 2004 15:02:09 +0000
>To: ronniec95@lineone.net
>Cc: caml-list@inria.fr
>Subject: Re: [Caml-list] CamlDL/Abstract pointers problem
>From: Richard Jones <rich@annexia.org>
>
>
>On Thu, Jan 29, 2004 at 01:06:39PM +0000, ronniec95@lineone.net wrote:
> All I want to do is just give Ocaml a pointer to something I've created
> in C and pass it back to other C functions later; don't want it to do
anything
> with it (including mo
>ing it around the ocaml heap).\

It's my understanding that you can't just pass a pointer back to OCaml
and cast it to a value.  OCaml will think that the pointer points to
an OCaml heap object, and hence try to examine / move / delete it in
the g
>rbage collector.  Instead you need to do one of two things:
either set the bottom bit to 1, in which case OCaml will treat it as a
plain integer, or else wrap it up in a box, as in the code below.  The
Abstract_tag tells the garbage collector not to
>look inside the box.

static value
Val_voidptr (void *ptr)
{
  CAMLparam0 ();
  CAMLlocal1 (rv);
  rv = alloc (1, Abstract_tag);
  Field(rv, 0) = (value) ptr;
  CAMLreturn (rv);
}

#define Voidptr_val(type,rv) ((type *) Field ((rv), 0))


>ou need to call Val_voidptr to wrap up your C void* pointer into an
OCaml box, and Voidptr_val to unwrap and get your pointer back again.

In any case you need to also add lots of calls to Gc.full_major () to
your code, which will trigger an early
>ore dump if there is some
memory corruption.

Rich.

--
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
If I have not seen as far as others, it is beca
>se I have been
standing in the footprints of giants.  -- from Usenet



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[art yerkes]

On Thu, 29 Jan 2004 15:52:10 +0000
ronniec95@lineone.net wrote:

> I understand that bit ok. But what I wanted to know is why CAMLIDL was not
> doing this for me... as I said in my original email, I'm quite happy handwriting
> all this stuff (Abstract_tag/Custom_tag all ok for me )- it works fine,
> but is quite dull!

You might try SWIG for this (http://www.swig.org/).  It will do the tedious part,
at least.
-- 
"Should array indices start at 0 or 1? My compromise of 0.5 was rejected
 without, I thought, proper consideration."
   - S. Kelly-Bootle

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[skaller]

On Fri, 2004-01-30 at 02:02, Richard Jones wrote:
> On Thu, Jan 29, 2004 at 01:06:39PM +0000, ronniec95@lineone.net wrote:
> > All I want to do is just give Ocaml a pointer to something I've created
> > in C and pass it back to other C functions later; don't want it to do anything
> > with it (including moving it around the ocaml heap).\
> 
> It's my understanding that you can't just pass a pointer back to OCaml
> and cast it to a value.  OCaml will think that the pointer points to
> an OCaml heap object, and hence try to examine / move / delete it in
> the garbage collector. 

Nope: RTFM:

                                  18.2
                                  The value type
All Caml objects are represented by the C type value, defined in the
include file caml/mlvalues.h, along with macros to manipulate values of
that type. An object of type value is either: 
      * an unboxed integer;
      * a pointer to a block inside the heap (such as the blocks
        allocated through one of the alloc_* functions below);
      * a pointer to an object outside the heap (e.g., a pointer to a
        block allocated by malloc, or to a C variable).

18.2.3 Pointers outside the heap

Any word-aligned pointer to an address outside the heap can be safely
cast to and from the type value. This includes pointers returned by
malloc, and pointers to C variables (of size at least one word) obtained
with the & operator.

[Note the manual is wrong here in that a C variable of word
size need not be word aligned .. it isn't necessary on x86 machines,
x86 doesn't require integers or addresses to be aligned..however
most C compilers do the alignment anyhow because it 
improves performance. I think floats do have to be aligned though,
and thus malloc must return an even address] 

-- 
John Max Skaller, mailto:skaller@tpg.com.au
snail:25/85c Wigram Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850. Checkout Felix: http://felix.sf.net




-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Richard Jones]

On Fri, Jan 30, 2004 at 06:04:41AM +1100, skaller wrote:
> 18.2.3 Pointers outside the heap

I'm pretty sure this bit of the manual is wrong, or at least I've not
been able to make it work.  I don't have any definite proof of this,
but my crashes stopped once I started wrapping up pointers in
Abstract_tag.

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
PTHRLIB is a library for writing small, efficient and fast servers in C.
HTTP, CGI, DBI, lightweight threads: http://www.annexia.org/freeware/pthrlib/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Alain.Frisch]

On Thu, 29 Jan 2004, Richard Jones wrote:

> I'm pretty sure this bit of the manual is wrong, or at least I've not
> been able to make it work.  I don't have any definite proof of this,
> but my crashes stopped once I started wrapping up pointers in
> Abstract_tag.

One important thing to be aware of when you use "foreign" pointers is
that the Caml GC heap can grow at any time and span over existing
pointers. So you have to be careful to null foreign pointers when you free
the block they're pointing to.

Reference: http://pauillac.inria.fr/~doligez/caml-guts/Fahndrich99.txt

-- Alain

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Richard Jones]

On Thu, Jan 29, 2004 at 08:55:16PM +0100, Alain.Frisch@ens.fr wrote:
> Reference: http://pauillac.inria.fr/~doligez/caml-guts/Fahndrich99.txt

Interesting.  Can someone who knows about the guts of OCaml comment on
this: Does OCaml store simply the start and extent of the "ML Heap"?
Or does it know about individual blocks of ML heap and use some sort
of tree structure to work out if a pointer points into the heap?

I could imagine a scenario like this:

+---------+
| ML      |
| Heap    |
|         |
|    ---------\
|         |   |
+---------+   |
              |
+---------+   |
| C       |<--/
| malloc  |
+---------+

+---------+
| More    |
| ML      |
| Heap    |
|         |
+---------+

If OCaml only knew about the start and extent of the ML heap, then it
might think that the pointer to the C-allocated space lies within the
ML heap.

Rich.

-- 
Richard Jones. http://www.annexia.org/ http://www.j-london.com/
Merjis Ltd. http://www.merjis.com/ - improving website return on investment
C2LIB is a library of basic Perl/STL-like types for C. Vectors, hashes,
trees, string funcs, pool allocator: http://www.annexia.org/freeware/c2lib/

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[David Brown]

On Thu, Jan 29, 2004 at 08:24:04PM +0000, Richard Jones wrote:
> On Thu, Jan 29, 2004 at 08:55:16PM +0100, Alain.Frisch@ens.fr wrote:
> > Reference: http://pauillac.inria.fr/~doligez/caml-guts/Fahndrich99.txt
> 
> Interesting.  Can someone who knows about the guts of OCaml comment on
> this: Does OCaml store simply the start and extent of the "ML Heap"?
> Or does it know about individual blocks of ML heap and use some sort
> of tree structure to work out if a pointer points into the heap?

Well, it uses a page table, but yes, it does do what you expect.

I have several applications that use types that are pointers outside of
the ocaml heap, and they work well.

Dave

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Jacques Garrigue]

From: Richard Jones <rich@annexia.org>
> On Thu, Jan 29, 2004 at 08:55:16PM +0100, Alain.Frisch@ens.fr wrote:
> > Reference: http://pauillac.inria.fr/~doligez/caml-guts/Fahndrich99.txt
> 
> Interesting.  Can someone who knows about the guts of OCaml comment on
> this: Does OCaml store simply the start and extent of the "ML Heap"?
> Or does it know about individual blocks of ML heap and use some sort
> of tree structure to work out if a pointer points into the heap?
> 
> I could imagine a scenario like this:
> 
> +---------+
> | ML      |
> | Heap    |
> |         |
> |    ---------\
> |         |   |
> +---------+   |
>               |
> +---------+   |
> | C       |<--/
> | malloc  |
> +---------+
> 
> +---------+
> | More    |
> | ML      |
> | Heap    |
> |         |
> +---------+
> 
> If OCaml only knew about the start and extent of the ML heap, then it
> might think that the pointer to the C-allocated space lies within the
> ML heap.

Fortunately, ocaml is clever enough to properly determine what is the
ML heap.
However, there is another scenario which could cause you problems:
if for some reason your C pointer get freed, and later the ML heap is
extended so that it includes the address of the pointer, then you may
have some pointers left on the ML which will now be seen as ML
pointers.

 +---------+
 | ML      |
 | Heap    |
 |         |
 |    ---------\
 |         |   |
 +---------+   |
               |
 +---------+   |
 | More ML |<--/
 | Heap    |
 +---------+

Some people have got problems with that.
So the safe rule is: only use "raw" C pointers (not wrapped in a
custom block) when you are sure the pointed block will not be freed.
But this is self-evident: if the pointed block is managed, then you
should probably have a custom block to cooperate with C memory
management anyway.

           Jacques Garrigue

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Brian Hurt]

On Fri, 30 Jan 2004, Jacques Garrigue wrote:

> However, there is another scenario which could cause you problems:
> if for some reason your C pointer get freed, and later the ML heap is
> extended so that it includes the address of the pointer, then you may
> have some pointers left on the ML which will now be seen as ML
> pointers.

ML doesn't memset memory it just allocated?  This may be a problem- I 
don't *think* the C standard requires malloc to memset the memory itself, 
which means you might be getting garbage memory.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Jacques Garrigue]

From: Brian Hurt <bhurt@spnz.org>
> On Fri, 30 Jan 2004, Jacques Garrigue wrote:
> 
> > However, there is another scenario which could cause you problems:
> > if for some reason your C pointer get freed, and later the ML heap is
> > extended so that it includes the address of the pointer, then you may
> > have some pointers left on the ML which will now be seen as ML
> > pointers.
> 
> ML doesn't memset memory it just allocated?  This may be a problem- I 
> don't *think* the C standard requires malloc to memset the memory itself, 
> which means you might be getting garbage memory.

That's not the problem: blocks used by ML are always initialized
anyway.

The trouble is that you may at some point in time pass to your ML
program a pointer to some C structure outside of the heap. Since it is
outside of the heap, the garbage collector does not attempt to visit
it, and this creates no problem.
But it may happen (if your structure is freed on the C side), that at
some time in the future a new page is added to the heap, containing
the location where was the C structure.  If your pointer on the ML
side is still around, this is now a pointer to somewhere in the heap,
and the GC will attempt to visit it.

Even if ML was to zero the page before using it, after putting some
ML objects in the page, your pointer can easily end up pointing in the
middle of some random data.

Jacques Garrigue

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[Brian Hurt]

On Sat, 31 Jan 2004, Jacques Garrigue wrote:

> The trouble is that you may at some point in time pass to your ML
> program a pointer to some C structure outside of the heap. Since it is
> outside of the heap, the garbage collector does not attempt to visit
> it, and this creates no problem.
> But it may happen (if your structure is freed on the C side), that at
> some time in the future a new page is added to the heap, containing
> the location where was the C structure.  If your pointer on the ML
> side is still around, this is now a pointer to somewhere in the heap,
> and the GC will attempt to visit it.

Ah.  OK.  That's reasonable- solving it in the Ocaml system would be way 
too hard to be worthwhile.

-- 
"Usenet is like a herd of performing elephants with diarrhea -- massive,
difficult to redirect, awe-inspiring, entertaining, and a source of
mind-boggling amounts of excrement when you least expect it."
                                - Gene Spafford 
Brian

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] CamlDL/Abstract pointers problem

[skaller]

On Fri, 2004-01-30 at 18:04, Brian Hurt wrote:
> On Fri, 30 Jan 2004, Jacques Garrigue wrote:
> 
> > However, there is another scenario which could cause you problems:
> > if for some reason your C pointer get freed, and later the ML heap is
> > extended so that it includes the address of the pointer, then you may
> > have some pointers left on the ML which will now be seen as ML
> > pointers.
> 
> ML doesn't memset memory it just allocated?  

It doesn't need to .. all values are initialised (in Ocaml).
When writing C/Ocaml glue, you have to be careful
that you *do* initialise all values before possibly triggering
the collector.. 

> This may be a problem- I 
> don't *think* the C standard requires malloc to memset the memory itself, 

it doesn't. The function which does do that is
called 'calloc'.

-- 
John Max Skaller, mailto:skaller@tpg.com.au
snail:25/85c Wigram Rd, Glebe, NSW 2037, Australia.
voice:61-2-9660-0850. Checkout Felix: http://felix.sf.net




-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners