A pair of "Interfacing with C" questions

A pair of "Interfacing with C" questions

[Robert Roessler]

1) in wrapping a large widget with multiple interfaces using 
"strings", I sometimes allow the widget to copy a C-string into a 
Caml-allocated string - INCLUDING "overwriting" the terminating zero 
byte with zero... I wanted to make sure this was not seen as a major 
problem.

As I understand the Caml runtime's use of strings (at least one zero 
byte always being present), this does not seem like a problem to me - 
and until memory protection granularity practices change a lot, there 
should not be any trouble there... :)

2) this is about "future-proofing" (at the source level) code which 
interfaces with external [foreign] components and needs to store and 
pass around "opaque pointers": what is the best base Caml data type to 
use?

I currently use

type opaque_ptr = int32

I assume the other choices include int64, nativeint, or even int.

int64 - good if you know you are dealing with 64-bit values, obviously 
not suitable in the general case.

nativeint - this looks promising, as long as you do not need to deal 
with a 64-bit-int/32-bit-pointer model... would it be safe to assume 
that in any 64-bit Caml implementation, ints/pointers/"values" will 
ALL be 64-bit?

int - there clearly is a restriction on using only pointers with the 
low bit set to zero; if you can stipulate that this is a non-issue, 
then you can just SET/CLEAR the low bit at the Caml/C boundary and 
then be able to safely store to and retrieve from Caml "value" fields 
(I think).

So... is the winner int or nativeint?  Or ?  And yes, I know that 
using int will avoid an extra allocation... but are there other 
considerations not represented here?

Any relevant insights will be appreciated. :)

Robert Roessler
roessler@rftp.com
http://www.rftp.com

Re: [Caml-list] A pair of "Interfacing with C" questions

[Robert Roessler]

dmitry grebeniuk wrote:

> Shalom, Robert.
> ...
> 2) int32 -- good enough.  No limitations and drawbacks of ints/values,
> but it is boxed.

Thanks, dmitry - but this breaks as soon as the software is recompiled 
on a 64-bit implementation (AFAICT). :)

> 3) custom blocks with size sizeof(yourpointertype).  I think it is the
> best alternative: you can attach finalization function (and others, if
> you will need) to blocks.  In this function you can: a) automatically
> deallocate C-side objects, b) write debug info about objects that are
> not closed properly from Caml-side.

Yes, I use custom blocks when their functionality is needed - but here 
I really just want "opaque pointers"... I only need to store them in a 
Caml "value" and be able to hand them back across the Caml/C interface 
on demand.

Robert Roessler
roessler@rftp.com
http://www.rftp.com

Re: [Caml-list] A pair of "Interfacing with C" questions

[Jacques Garrigue]

From: Robert Roessler <roessler@rftp.com>

> 1) in wrapping a large widget with multiple interfaces using 
> "strings", I sometimes allow the widget to copy a C-string into a 
> Caml-allocated string - INCLUDING "overwriting" the terminating zero 
> byte with zero... I wanted to make sure this was not seen as a major 
> problem.

No problem currently: for C compatibility all ocaml strings have a 0
after their end. Nobody can give "future" guarantees, but if there is
a change at this level, I assume this would be well publicised.

> 2) this is about "future-proofing" (at the source level) code which 
> interfaces with external [foreign] components and needs to store and 
> pass around "opaque pointers": what is the best base Caml data type to 
> use?
> 
> I currently use
> 
> type opaque_ptr = int32
> 
> I assume the other choices include int64, nativeint, or even int.

If you look at mlvalues.h you will see that nativeint is define as
long int. So this should be the natural format for a pointer.

The other option is to use store them in an ocaml int, without any
conversion, as 4-byte aligned pointers outside of the heap are ignored
by the GC. But this has some pitfalls, as some dangling pointers may
end up pointing to a newly allocated heap area. This is also less
"future proof", as the representation of ocaml integers might change
someday.

Another advantage of nativeint is that they are custom block, so you
just have to change the allocation to make them GC-aware. This is why
I converted all pointers to custom-format blocks in lablgtk.

> nativeint - this looks promising, as long as you do not need to deal 
> with a 64-bit-int/32-bit-pointer model... would it be safe to assume 
> that in any 64-bit Caml implementation, ints/pointers/"values" will 
> ALL be 64-bit?

This is certainly currently the case, and without a radical change in
the approach to polymorphism, all "basic" ocaml types must have the
same size, and for practical reasons it is good that machine pointers
fit in this size. I suppose this does not exclude compiling ocaml in
32-bit mode on 64-bit machines that support such a mode, but even in this
case this supposes that (32-bit) pointers fit in an ocaml memory unit.

Jacques Garrigue

Re: [Caml-list] A pair of "Interfacing with C" questions

[malc]

On Thu, 21 Jul 2005, Jacques Garrigue wrote:

> From: Robert Roessler <roessler@rftp.com>
>
>> 1) in wrapping a large widget with multiple interfaces using
>> "strings", I sometimes allow the widget to copy a C-string into a
>> Caml-allocated string - INCLUDING "overwriting" the terminating zero
>> byte with zero... I wanted to make sure this was not seen as a major
>> problem.
>
> No problem currently: for C compatibility all ocaml strings have a 0
> after their end. Nobody can give "future" guarantees, but if there is
> a change at this level, I assume this would be well publicised.
>
>> 2) this is about "future-proofing" (at the source level) code which
>> interfaces with external [foreign] components and needs to store and
>> pass around "opaque pointers": what is the best base Caml data type to
>> use?
>>
>> I currently use
>>
>> type opaque_ptr = int32
>>
>> I assume the other choices include int64, nativeint, or even int.
>
> If you look at mlvalues.h you will see that nativeint is define as
> long int. So this should be the natural format for a pointer.

I don't think so, Win64 ABI has sizeof(long) == 4 and sizeof(void *) == 8.

<snip>

-- 
mailto:malc@pulsesoft.com

Re: [Caml-list] A pair of "Interfacing with C" questions

[Jacques Garrigue]

From: malc <malc@pulsesoft.com>

> >> I assume the other choices include int64, nativeint, or even int.
> >
> > If you look at mlvalues.h you will see that nativeint is define as
> > long int. So this should be the natural format for a pointer.
> 
> I don't think so, Win64 ABI has sizeof(long) == 4 and sizeof(void *) == 8.

Aargh, they seem to like to break everything!
Anyway, I suppose it just means that ocaml would have to define value
as a long long int on Win64, to make it behave like other
architectures. 
The sources are full of casts from value to (value *) and back.
Rather, this is going to be a pain to port many C libraries, that
assume that you can convert between a long and a pointer and back.

Jacques Garrigue

Re: [Caml-list] A pair of "Interfacing with C" questions

[Robert Roessler]

Jacques Garrigue wrote:
> From: malc <malc@pulsesoft.com>
> 
>>>>I assume the other choices include int64, nativeint, or even int.
>>>
>>>If you look at mlvalues.h you will see that nativeint is define as
>>>long int. So this should be the natural format for a pointer.
>>
>>I don't think so, Win64 ABI has sizeof(long) == 4 and sizeof(void *) == 8.
> 
> 
> Aargh, they seem to like to break everything!
> Anyway, I suppose it just means that ocaml would have to define value
> as a long long int on Win64, to make it behave like other
> architectures. 
> The sources are full of casts from value to (value *) and back.
> Rather, this is going to be a pain to port many C libraries, that
> assume that you can convert between a long and a pointer and back.

I was worried about something like the Win64 silliness... ;)

Said silliness aside, I will go ahead and switch to nativeint - thanks.

Robert Roessler
roessler@rftp.com
http://www.rftp.com

Re: [Caml-list] A pair of "Interfacing with C" questions

[Alexander S. Usov]

On Friday 22 July 2005 07:26, Robert Roessler wrote:
> Jacques Garrigue wrote:
> > From: malc <malc@pulsesoft.com>
> >
> >>>>I assume the other choices include int64, nativeint, or even int.
> >>>
> >>>If you look at mlvalues.h you will see that nativeint is define as
> >>>long int. So this should be the natural format for a pointer.
> >>
> >>I don't think so, Win64 ABI has sizeof(long) == 4 and sizeof(void *) ==
> >> 8.
> >
> > Aargh, they seem to like to break everything!
> > Anyway, I suppose it just means that ocaml would have to define value
> > as a long long int on Win64, to make it behave like other
> > architectures.
> > The sources are full of casts from value to (value *) and back.
> > Rather, this is going to be a pain to port many C libraries, that
> > assume that you can convert between a long and a pointer and back.
>
> I was worried about something like the Win64 silliness... ;)
>
> Said silliness aside, I will go ahead and switch to nativeint - thanks.

According to 
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/win64/win64/abstract_data_models.asp
they have chosen LLP64 model, while almost everybody else are using LP64. 
Quite funny choice I  should say.

But from the other point of view -- isn't it possible to make the compiler to 
complain hard when you are trying to convert int/long to pointer?



-- 
Best regards,
  Alexander.

Re: [Caml-list] A pair of "Interfacing with C" questions

[Xavier Leroy]

>> If you look at mlvalues.h you will see that nativeint is define as
>> long int. So this should be the natural format for a pointer.
>
>
> I don't think so, Win64 ABI has sizeof(long) == 4 and sizeof(void *) == 8.

... which is why a port of OCaml to Win64 would use long long (or,
better, intptr_t, if available) for the C type "value" and the Caml
type "nativeint".

In other terms, OCaml values are integers big enough to hold a
pointer, and the type nativeint has (by construction) the same size as
a value.

- Xavier Leroy