[Caml-list] Finding "lost" references to OCaml heap values

[Caml-list] Finding "lost" references to OCaml heap values

[Richard W.M. Jones]

I guess I have two questions:

(1) Is calling Gc.compact () guaranteed to call the finalizer of any
object which is no longer reachable, under all circumstances?  Or
would there be some case where it wouldn't be called?

(2) I have a large mixed OCaml / C program[a] where somehow calling
Gc.compact isn't calling the destructor of a (very) large object.
Manual code inspection has not revealed anything so far --
superficially it appears we are not holding any references to the
object.  Is there any method / library / tool that can inspect the
OCaml heap and find references to an object?

Rich.

[a] https://github.com/libguestfs/libguestfs/tree/master/v2v

-- 
Richard Jones
Red Hat

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Richard W.M. Jones]

On Tue, Oct 06, 2015 at 02:43:42PM +0100, Richard W.M. Jones wrote:
> 
> I guess I have two questions:
> 
> (1) Is calling Gc.compact () guaranteed to call the finalizer of any
> object which is no longer reachable, under all circumstances?  Or
> would there be some case where it wouldn't be called?
> 
> (2) I have a large mixed OCaml / C program[a] where somehow calling
> Gc.compact isn't calling the destructor of a (very) large object.
> Manual code inspection has not revealed anything so far --
> superficially it appears we are not holding any references to the
> object.  Is there any method / library / tool that can inspect the
> OCaml heap and find references to an object?

Always good to explain these things, because the act of explaining it
has allowed me to work out why (2) is happening now.

The reason is because I was registering a global root from the C heap
pointing to the handle, and of course this prevents the handle from
being unreferenced.

This does, however, raise another question:

(3) I want to have a C heap 'value' pointing to an OCaml value, in
such a way that if the OCaml value moves around, the C value gets
updated to point to the new location.  I was using a global root
for this purpose, but it seems like global roots really have two
purposes:

  (i) To keep the C value updated if the OCaml value moves.

  (ii) To act as a global root, preventing the value from being freed.

Is there a "weak" global root, that has property (i) but not property (ii)?

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Gerd Stolpmann]

[-- Attachment #1: Type: text/plain, Size: 2565 bytes --]

Am Dienstag, den 06.10.2015, 15:16 +0100 schrieb Richard W.M. Jones:
> On Tue, Oct 06, 2015 at 02:43:42PM +0100, Richard W.M. Jones wrote:
> > 
> > I guess I have two questions:
> > 
> > (1) Is calling Gc.compact () guaranteed to call the finalizer of any
> > object which is no longer reachable, under all circumstances?  Or
> > would there be some case where it wouldn't be called?
> > 
> > (2) I have a large mixed OCaml / C program[a] where somehow calling
> > Gc.compact isn't calling the destructor of a (very) large object.
> > Manual code inspection has not revealed anything so far --
> > superficially it appears we are not holding any references to the
> > object.  Is there any method / library / tool that can inspect the
> > OCaml heap and find references to an object?
> 
> Always good to explain these things, because the act of explaining it
> has allowed me to work out why (2) is happening now.
> 
> The reason is because I was registering a global root from the C heap
> pointing to the handle, and of course this prevents the handle from
> being unreferenced.
> 
> This does, however, raise another question:
> 
> (3) I want to have a C heap 'value' pointing to an OCaml value, in
> such a way that if the OCaml value moves around, the C value gets
> updated to point to the new location.  I was using a global root
> for this purpose, but it seems like global roots really have two
> purposes:
> 
>   (i) To keep the C value updated if the OCaml value moves.
> 
>   (ii) To act as a global root, preventing the value from being freed.
> 
> Is there a "weak" global root, that has property (i) but not property (ii)?

There is nothing like that in the C API of the OCaml FFI. What you can
do is to keep only a symbolic reference to the OCaml value from the C
struct (i.e. if it is the k-th such value just store the number k). The
mapping from k to the OCaml value can be done with a weak hash table.
For finalizing this you need to use Gc.finalize.

All in all this is quite complicated. I guess it's only worth it if the
memory management really needs to be fully-automatic.

Gerd

> Rich.
> 
> -- 
> Richard Jones
> Red Hat
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
My OCaml site:          http://www.camlcity.org
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
------------------------------------------------------------


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 2854 bytes --]

Sorry for the rather naive suggestion, but wouldn't it be possible to
create a weak pointer in the OCaml world, and have the C value refer to
that weak pointer?

On Tue, Oct 6, 2015 at 5:09 PM, Gerd Stolpmann <info@gerd-stolpmann.de>
wrote:

> Am Dienstag, den 06.10.2015, 15:16 +0100 schrieb Richard W.M. Jones:
> > On Tue, Oct 06, 2015 at 02:43:42PM +0100, Richard W.M. Jones wrote:
> > >
> > > I guess I have two questions:
> > >
> > > (1) Is calling Gc.compact () guaranteed to call the finalizer of any
> > > object which is no longer reachable, under all circumstances?  Or
> > > would there be some case where it wouldn't be called?
> > >
> > > (2) I have a large mixed OCaml / C program[a] where somehow calling
> > > Gc.compact isn't calling the destructor of a (very) large object.
> > > Manual code inspection has not revealed anything so far --
> > > superficially it appears we are not holding any references to the
> > > object.  Is there any method / library / tool that can inspect the
> > > OCaml heap and find references to an object?
> >
> > Always good to explain these things, because the act of explaining it
> > has allowed me to work out why (2) is happening now.
> >
> > The reason is because I was registering a global root from the C heap
> > pointing to the handle, and of course this prevents the handle from
> > being unreferenced.
> >
> > This does, however, raise another question:
> >
> > (3) I want to have a C heap 'value' pointing to an OCaml value, in
> > such a way that if the OCaml value moves around, the C value gets
> > updated to point to the new location.  I was using a global root
> > for this purpose, but it seems like global roots really have two
> > purposes:
> >
> >   (i) To keep the C value updated if the OCaml value moves.
> >
> >   (ii) To act as a global root, preventing the value from being freed.
> >
> > Is there a "weak" global root, that has property (i) but not property
> (ii)?
>
> There is nothing like that in the C API of the OCaml FFI. What you can
> do is to keep only a symbolic reference to the OCaml value from the C
> struct (i.e. if it is the k-th such value just store the number k). The
> mapping from k to the OCaml value can be done with a weak hash table.
> For finalizing this you need to use Gc.finalize.
>
> All in all this is quite complicated. I guess it's only worth it if the
> memory management really needs to be fully-automatic.
>
> Gerd
>
> > Rich.
> >
> > --
> > Richard Jones
> > Red Hat
> >
>
> --
> ------------------------------------------------------------
> Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
> My OCaml site:          http://www.camlcity.org
> Contact details:        http://www.camlcity.org/contact.html
> Company homepage:       http://www.gerd-stolpmann.de
> ------------------------------------------------------------
>
>

[-- Attachment #2: Type: text/html, Size: 3890 bytes --]

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Richard W.M. Jones]

On Tue, Oct 06, 2015 at 05:17:48PM +0200, Gabriel Scherer wrote:
> Sorry for the rather naive suggestion, but wouldn't it be possible to
> create a weak pointer in the OCaml world, and have the C value refer to
> that weak pointer?

It seems like that should work.

I managed to get rid of the global root by thinking a bit harder about
the code, so I solved it another way.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Maxime Ransan (BLOOMBERG/ 731 LEX)]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

Just a hint about why the destructor is not called is that when using caml_alloc_custom at  https://github.com/libguestfs/libguestfs/blob/master/v2v/xml-c.c#L139, you are setting the used parameter to 0.  

As mentioned in http://caml.inria.fr/pub/docs/manual-ocaml-4.00/manual033.html#toc150 you can increase the ratio. A quick check would be to use a (used = 1, max = 1).

From: rich@annexia.org At: Oct  6 2015 09:43:53
To: caml-list@inria.fr
Subject: Re:[Caml-list] Finding "lost" references to OCaml heap values


I guess I have two questions:

(1) Is calling Gc.compact () guaranteed to call the finalizer of any
object which is no longer reachable, under all circumstances?  Or
would there be some case where it wouldn't be called?

(2) I have a large mixed OCaml / C program[a] where somehow calling
Gc.compact isn't calling the destructor of a (very) large object.
Manual code inspection has not revealed anything so far --
superficially it appears we are not holding any references to the
object.  Is there any method / library / tool that can inspect the
OCaml heap and find references to an object?

Rich.

[a] https://github.com/libguestfs/libguestfs/tree/master/v2v

-- 
Richard Jones
Red Hat

-- 
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs



[-- Attachment #2: Type: text/html, Size: 3370 bytes --]

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Richard W.M. Jones]

On Tue, Oct 06, 2015 at 01:57:52PM -0000, Maxime Ransan (BLOOMBERG/ 731 LEX) wrote:
> Just a hint about why the destructor is not called is that when using caml_alloc_custom at  https://github.com/libguestfs/libguestfs/blob/master/v2v/xml-c.c#L139, you are setting the used parameter to 0.  
> 
> As mentioned in http://caml.inria.fr/pub/docs/manual-ocaml-4.00/manual033.html#toc150 you can increase the ratio. A quick check would be to use a (used = 1, max = 1).

I guess this probably doesn't have anything to do with this
specific problem, since I'm calling the GC explicitly.

However you're certainly correct that we don't pay any attention to
passing decent values for caml_alloc_custom used/max.

One reason is that almost all OCaml programs we have are relatively
short running, so the pace of GC isn't much of an issue.  In fact, we
didn't notice that Gc.compact () was having "no effect" - it was only
noticed by an outside contributor who was doing a particularly long
v2v conversion and investigated further.

Thanks,

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] Finding "lost" references to OCaml heap values

[Richard W.M. Jones]

On Tue, Oct 06, 2015 at 04:53:11PM +0100, Richard W.M. Jones wrote:
> On Tue, Oct 06, 2015 at 01:57:52PM -0000, Maxime Ransan (BLOOMBERG/ 731 LEX) wrote:
> > Just a hint about why the destructor is not called is that when using caml_alloc_custom at  https://github.com/libguestfs/libguestfs/blob/master/v2v/xml-c.c#L139, you are setting the used parameter to 0.  
> > 
> > As mentioned in http://caml.inria.fr/pub/docs/manual-ocaml-4.00/manual033.html#toc150 you can increase the ratio. A quick check would be to use a (used = 1, max = 1).
> 
> I guess this probably doesn't have anything to do with this
> specific problem, since I'm calling the GC explicitly.
> 
> However you're certainly correct that we don't pay any attention to
> passing decent values for caml_alloc_custom used/max.

I was going to add here:

... because it's hard to choose good values!  What's a good 'used' for
a libguestfs handle that might consume 100 MB?  And what's a good
'max' (maybe total physical RAM)?

Also, suppose I choose a large used/max (1/1 for example), is there a
danger that the major GC will do too much work?  The manual is unclear
on whether a bad choice of used/max can cause negative effects.

Rich.

-- 
Richard Jones
Red Hat