[Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

["Martin R. Neuhäußer"]

Dear all,

after an intense week of debugging some large C-bindings, I presume to have found a bug in OCaml’s garbage collection code or in its Weak module. 
To summarize, it seems as if OCaml’s garbage collector and its Weak module sometimes release a custom block (by calling its finalizer) too early, i.e. when it is still reachable.

I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail. Therefore I’d like to double-check some of the assumptions that I have made when writing my C-stubs. Any corrections are highly welcome:
1. Custom blocks may be moved around in memory by the GC, but they are never duplicated.
2. The finalizer for each custom block that is allocated by caml_alloc_custom is called at most once.
3. The finalizer is never called for blocks that are still alive; stated otherwise, a block that has been finalized can never been presented as a value to a C-stub anymore.

There is a small example program available on github: https://github.com/martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violated.

The idea is as follows:
Each custom block that the example allocates contains a pointer to structure that is malloc’ed outside the OCaml heap; each custom block uniquely refers to such a structure.
Upon creation of a custom block, a special flag-value is stored in its corresponding (newly created) structure to indicate that the block is alive. Once a custom block’s finalizer is called, the flag is changed to indicate finalization of the block. I assume that after the finalizer returns, the custom block is unreachable (and perhaps its memory reclaimed by the GC); however, for debugging, I intentionally keep the custom block’s structure allocated on the unmanaged heap. Whenever a custom block is presented to the C-stubs, they check that the corresponding flag-value within its corresponding non-managed heap structure indicates liveness of the block. As it turns out, this invariant is violated, if the Weak module and the GC are stressed.

Of course, it might very well be that there is a bug in my C-stubs that is responsible for that behavior. I tested with different versions of OCaml, with different outcomes: The example program terminates correctly with OCaml 3.12.1, 4.00.0, 4.00.1, 4.01.0, and 4.03.0+beta1 whereas it crashes due to the above error when compiled with OCaml 4.02.0, 4.02.1, 4.02.2, 4.02.3, 4.03.0+beta2, and the released 4.03.0. The tests were run on 64bit Debian Linux and 64bit MacOS X.

I'd very much appreciate any help, hints to bugs in my code, or requests for clarification.

Thanks and best regards,
Martin

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Gabriel Scherer]

[-- Attachment #1: Type: text/plain, Size: 4105 bytes --]

> 3. The finalizer is never called for blocks that are still alive; stated
> otherwise, a block that has been finalized can never been presented as a
> value to a C-stub anymore.


I think that this does not hold in general: the finalizer should only be
called when there is no reference to the value around, *but* the finalizer
action can create a new reference (for example you may insert the value
into a (live) global cache), so the value may be live after the finalizer
has been called -- and be sent to a C stub, etc.

This remark may not apply to your actual reproduction case (your finalizer
shouldn't make the value reachable from the OCaml heap).

(There were changes between the 4.03 development cycle that would explain
the difference in behavior between beta1 and beta2, see in particular
http://caml.inria.fr/mantis/view.php?id=7210 and
https://github.com/ocaml/ocaml/pull/22 ; but I'll leave memory-runtime
experts comment on that.)

On Fri, Jun 17, 2016 at 2:09 PM, "Martin R. Neuhäußer" <post@marneu.com>
wrote:

> Dear all,
>
> after an intense week of debugging some large C-bindings, I presume to
> have found a bug in OCaml’s garbage collection code or in its Weak module.
> To summarize, it seems as if OCaml’s garbage collector and its Weak module
> sometimes release a custom block (by calling its finalizer) too early, i.e.
> when it is still reachable.
>
> I hesitate a bit before opening an „official“ issue on Mantis as I might
> very well overlook some detail. Therefore I’d like to double-check some of
> the assumptions that I have made when writing my C-stubs. Any corrections
> are highly welcome:
> 1. Custom blocks may be moved around in memory by the GC, but they are
> never duplicated.
> 2. The finalizer for each custom block that is allocated by
> caml_alloc_custom is called at most once.
> 3. The finalizer is never called for blocks that are still alive; stated
> otherwise, a block that has been finalized can never been presented as a
> value to a C-stub anymore.
>
> There is a small example program available on github:
> https://github.com/martin-neuhaeusser/ocaml_bug where the above
> assumptions seem to be violated.
>
> The idea is as follows:
> Each custom block that the example allocates contains a pointer to
> structure that is malloc’ed outside the OCaml heap; each custom block
> uniquely refers to such a structure.
> Upon creation of a custom block, a special flag-value is stored in its
> corresponding (newly created) structure to indicate that the block is
> alive. Once a custom block’s finalizer is called, the flag is changed to
> indicate finalization of the block. I assume that after the finalizer
> returns, the custom block is unreachable (and perhaps its memory reclaimed
> by the GC); however, for debugging, I intentionally keep the custom block’s
> structure allocated on the unmanaged heap. Whenever a custom block is
> presented to the C-stubs, they check that the corresponding flag-value
> within its corresponding non-managed heap structure indicates liveness of
> the block. As it turns out, this invariant is violated, if the Weak module
> and the GC are stressed.
>
> Of course, it might very well be that there is a bug in my C-stubs that is
> responsible for that behavior. I tested with different versions of OCaml,
> with different outcomes: The example program terminates correctly with
> OCaml 3.12.1, 4.00.0, 4.00.1, 4.01.0, and 4.03.0+beta1 whereas it crashes
> due to the above error when compiled with OCaml 4.02.0, 4.02.1, 4.02.2,
> 4.02.3, 4.03.0+beta2, and the released 4.03.0. The tests were run on 64bit
> Debian Linux and 64bit MacOS X.
>
> I'd very much appreciate any help, hints to bugs in my code, or requests
> for clarification.
>
> Thanks and best regards,
> Martin
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

[-- Attachment #2: Type: text/html, Size: 5103 bytes --]

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Török Edwin]

On 06/17/2016 09:09 PM, "Martin R. Neuhäußer" wrote:
> Dear all,
> 
> after an intense week of debugging some large C-bindings, I presume to have found a bug in OCaml’s garbage collection code or in its Weak module. 
> To summarize, it seems as if OCaml’s garbage collector and its Weak module sometimes release a custom block (by calling its finalizer) too early, i.e. when it is still reachable.
> 
> I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail. Therefore I’d like to double-check some of the assumptions that I have made when writing my C-stubs. Any corrections are highly welcome:
> 1. Custom blocks may be moved around in memory by the GC, but they are never duplicated.
> 2. The finalizer for each custom block that is allocated by caml_alloc_custom is called at most once.
> 3. The finalizer is never called for blocks that are still alive; stated otherwise, a block that has been finalized can never been presented as a value to a C-stub anymore.

Shallow copies don't seem very safe in the presence of out-of-OCaml-heap C pointers [*].
Perhaps Weak.get_copy should raise if it encounters a Custom_tag?

The custom value may contain C pointers, that may get invalidated or changed if the custom value has a finalizer (as in the testcase):
When the original weak value has no more references it gets the finalizer invoked, and gets garbage collected
 (in this case it changes unique_id to c_data_invalid_id, but it might as well free it instead causing a crash).

Meanwhile the copy made through Weak.get_copy still lives, and shares the copied custom value (pointer) from the weak value that got finalized.
Operations on the copied value (e.g. comparison/hashing) will try to access the custom value, which points to c_data_invalid_id.


> There is a small example program available on github: https://github.com/martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violated.

[*] AFAICT a shallow copy is created by WeakDummySet.find -> Weak.get_copy

Best regards,
-- 
Edwin Török | Co-founder and Lead Developer

Skylable open-source object storage: reliable, fast, secure
http://www.skylable.com

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

["Martin R. Neuhäußer"]

[-- Attachment #1: Type: text/plain, Size: 3755 bytes --]

Thanks for the good point! And yes, the weak sets use Weak.get_copy internally.

However, the exact semantics of Weak.get_copy is still a bit unclear to me. It creates a „shallow copy“ of a value that might independently be garbage collected. If that shallow copy can survive the finalization of its source, that might cause the problem. Are there any restrictions enforced on values obtained by Weak.get_copy? Or could this even be misused to create arbitrary copies of custom blocks that propagate anywhere?

I will try to extend the example such that maybe it triggers the problem outside the compare and hash functions, i.e. not in one of the functions that the weak set code might expose to a shallow copy.

Nevertheless, the behavior seems to have changes somewhere between OCaml 4.01.1 and 4.02.0. And the Mantis tickets 7161 and 7157 seem to affect OCaml’s behavior, as well. That might explain that the testcase succeeds with the first beta of OCaml 4.03.0…

Best,
Martin

> Am 17.06.2016 um 21:41 schrieb Török Edwin <edwin+ml-ocaml@etorok.net>:
> 
> On 06/17/2016 09:09 PM, "Martin R. Neuhäußer" wrote:
>> Dear all,
>> 
>> after an intense week of debugging some large C-bindings, I presume to have found a bug in OCaml’s garbage collection code or in its Weak module.
>> To summarize, it seems as if OCaml’s garbage collector and its Weak module sometimes release a custom block (by calling its finalizer) too early, i.e. when it is still reachable.
>> 
>> I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail. Therefore I’d like to double-check some of the assumptions that I have made when writing my C-stubs. Any corrections are highly welcome:
>> 1. Custom blocks may be moved around in memory by the GC, but they are never duplicated.
>> 2. The finalizer for each custom block that is allocated by caml_alloc_custom is called at most once.
>> 3. The finalizer is never called for blocks that are still alive; stated otherwise, a block that has been finalized can never been presented as a value to a C-stub anymore.
> 
> Shallow copies don't seem very safe in the presence of out-of-OCaml-heap C pointers [*].
> Perhaps Weak.get_copy should raise if it encounters a Custom_tag?
> 
> The custom value may contain C pointers, that may get invalidated or changed if the custom value has a finalizer (as in the testcase):
> When the original weak value has no more references it gets the finalizer invoked, and gets garbage collected
> (in this case it changes unique_id to c_data_invalid_id, but it might as well free it instead causing a crash).
> 
> Meanwhile the copy made through Weak.get_copy still lives, and shares the copied custom value (pointer) from the weak value that got finalized.
> Operations on the copied value (e.g. comparison/hashing) will try to access the custom value, which points to c_data_invalid_id.
> 
> 
>> There is a small example program available on github: https://github.com/martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violated.
> 
> [*] AFAICT a shallow copy is created by WeakDummySet.find -> Weak.get_copy
> 
> Best regards,
> --
> Edwin Török | Co-founder and Lead Developer
> 
> Skylable open-source object storage: reliable, fast, secure
> http://www.skylable.com
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

--
Martin Neuhäußer							Phone: +49 (911) 49051223
Normannenstraße 15						Mobile: +49 (172) 8966488
D-90461Nürnberg							GnuPG: 0x16FDB298


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 204 bytes --]

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Josh Berdine]

On Fri, Jun 17 2016, Martin R. Neuhäußer wrote:

> Thanks for the good point! And yes, the weak sets use Weak.get_copy internally.

I have also run into this when writing C bindings for a library with ref-counted memory management, where the OCaml finalizers decremented the C counts.  Since the functions in the structure passed to Weak.Make can be passed shallow copies of possibly-dead values, these functions need to be prepared for the underlying C values to have already been finalized.  That's tough to deal with.  (I ended up just letting Z3 hash-cons it's own terms, and kept pointers into Z3 out of my own hash-consed terms.)

> However, the exact semantics of Weak.get_copy is still a bit unclear to me. It creates a „shallow copy“ of a value that might independently be garbage collected. If that shallow copy can survive the finalization of its source, that might cause the problem. Are there any restrictions enforced on values obtained by Weak.get_copy? Or could this even be misused to create arbitrary copies of custom blocks that propagate anywhere?

If the functions passed to Weak.Make store the shallow copy into some global state, it seems that they could live on without constraint.

Cheers, Josh

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

["Martin R. Neuhäußer"]

[-- Attachment #1: Type: text/plain, Size: 3651 bytes --]

> 
> Am 17.06.2016 um 21:41 schrieb Török Edwin <edwin+ml-ocaml@etorok.net>:
> 
> On 06/17/2016 09:09 PM, "Martin R. Neuhäußer" wrote:
>> Dear all,
>> 
>> after an intense week of debugging some large C-bindings, I presume to have found a bug in OCaml’s garbage collection code or in its Weak module.
>> To summarize, it seems as if OCaml’s garbage collector and its Weak module sometimes release a custom block (by calling its finalizer) too early, i.e. when it is still reachable.
>> 
>> I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail. Therefore I’d like to double-check some of the assumptions that I have made when writing my C-stubs. Any corrections are highly welcome:
>> 1. Custom blocks may be moved around in memory by the GC, but they are never duplicated.
>> 2. The finalizer for each custom block that is allocated by caml_alloc_custom is called at most once.
>> 3. The finalizer is never called for blocks that are still alive; stated otherwise, a block that has been finalized can never been presented as a value to a C-stub anymore.
> 
> Shallow copies don't seem very safe in the presence of out-of-OCaml-heap C pointers [*].
> Perhaps Weak.get_copy should raise if it encounters a Custom_tag?
Just forbidding to use custom blocks in weak sets is harsh, isn’t it? For example, we are relying on SMT solvers and use weak sets on the OCaml side to do a lightweight hash-consing (actually, we only want to avoid having duplicate term representations in OCaml).
Moreover, I suspect it might be difficult to detect situations that involve custom blocks, as a custom block may occur deeply nested in a structured block that is stored in a weak set...
> 
> The custom value may contain C pointers, that may get invalidated or changed if the custom value has a finalizer (as in the testcase):
> When the original weak value has no more references it gets the finalizer invoked, and gets garbage collected
> (in this case it changes unique_id to c_data_invalid_id, but it might as well free it instead causing a crash).
> 
> Meanwhile the copy made through Weak.get_copy still lives, and shares the copied custom value (pointer) from the weak value that got finalized.
> Operations on the copied value (e.g. comparison/hashing) will try to access the custom value, which points to c_data_invalid_id.
I uploaded a second example `gcbug2.ml` which crashes as well but this time, it is designed not to check the „validity“ of custom blocks while a Weak.find operation is ongoing. Actually, in my experiments, this version aborts within the C-layer finalizer when trying to finalize a custom block that has been finalized before. So something like a double-free is happening here, and outside the scope of Weak.find.
If the behavior is related to the shallow copies created by Weak.get_copy, is it the case that they are finalized as well, and independently of their source custom block?
> 
> 
>> There is a small example program available on github: https://github.com/martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violated.
> 
> [*] AFAICT a shallow copy is created by WeakDummySet.find -> Weak.get_copy
> 
> Best regards,
> --
> Edwin Török | Co-founder and Lead Developer
> 
> Skylable open-source object storage: reliable, fast, secure
> http://www.skylable.com
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 204 bytes --]

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

["Martin R. Neuhäußer"]

[-- Attachment #1: Type: text/plain, Size: 4480 bytes --]


> Am 17.06.2016 um 21:23 schrieb Gabriel Scherer <gabriel.scherer@gmail.com>:
> 
> 
> 3. The finalizer is never called for blocks that are still alive; stated otherwise, a block that has been finalized can never been presented as a value to a C-stub anymore.
> 
> I think that this does not hold in general: the finalizer should only be called when there is no reference to the value around, *but* the finalizer action can create a new reference (for example you may insert the value into a (live) global cache), so the value may be live after the finalizer has been called -- and be sent to a C stub, etc.
It is definitely true that finalizers that are registered with Gc.finalise may generate new values (or make the value live again).
But I do not see how to create a new value within a finalizer that is attached to a custom block on the C-layer.
As the manual states, one must neither trigger a garbage collection in such a finalizer, nor call-back into OCaml code.

> 
> This remark may not apply to your actual reproduction case (your finalizer shouldn't make the value reachable from the OCaml heap).
> 
> (There were changes between the 4.03 development cycle that would explain the difference in behavior between beta1 and beta2, see in particular http://caml.inria.fr/mantis/view.php?id=7210 and https://github.com/ocaml/ocaml/pull/22 ; but I'll leave memory-runtime experts comment on that.)
> 
> On Fri, Jun 17, 2016 at 2:09 PM, "Martin R. Neuhäußer" <post@marneu.com> wrote:
> Dear all,
> 
> after an intense week of debugging some large C-bindings, I presume to have found a bug in OCaml’s garbage collection code or in its Weak module.
> To summarize, it seems as if OCaml’s garbage collector and its Weak module sometimes release a custom block (by calling its finalizer) too early, i.e. when it is still reachable.
> 
> I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail. Therefore I’d like to double-check some of the assumptions that I have made when writing my C-stubs. Any corrections are highly welcome:
> 1. Custom blocks may be moved around in memory by the GC, but they are never duplicated.
> 2. The finalizer for each custom block that is allocated by caml_alloc_custom is called at most once.
> 3. The finalizer is never called for blocks that are still alive; stated otherwise, a block that has been finalized can never been presented as a value to a C-stub anymore.
> 
> There is a small example program available on github: https://github.com/martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violated.
> 
> The idea is as follows:
> Each custom block that the example allocates contains a pointer to structure that is malloc’ed outside the OCaml heap; each custom block uniquely refers to such a structure.
> Upon creation of a custom block, a special flag-value is stored in its corresponding (newly created) structure to indicate that the block is alive. Once a custom block’s finalizer is called, the flag is changed to indicate finalization of the block. I assume that after the finalizer returns, the custom block is unreachable (and perhaps its memory reclaimed by the GC); however, for debugging, I intentionally keep the custom block’s structure allocated on the unmanaged heap. Whenever a custom block is presented to the C-stubs, they check that the corresponding flag-value within its corresponding non-managed heap structure indicates liveness of the block. As it turns out, this invariant is violated, if the Weak module and the GC are stressed.
> 
> Of course, it might very well be that there is a bug in my C-stubs that is responsible for that behavior. I tested with different versions of OCaml, with different outcomes: The example program terminates correctly with OCaml 3.12.1, 4.00.0, 4.00.1, 4.01.0, and 4.03.0+beta1 whereas it crashes due to the above error when compiled with OCaml 4.02.0, 4.02.1, 4.02.2, 4.02.3, 4.03.0+beta2, and the released 4.03.0. The tests were run on 64bit Debian Linux and 64bit MacOS X.
> 
> I'd very much appreciate any help, hints to bugs in my code, or requests for clarification.
> 
> Thanks and best regards,
> Martin
> 
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
> 


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 204 bytes --]

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Leo White]

> Shallow copies don't seem very safe in the presence of out-of-OCaml-heap C pointers [*].
> Perhaps Weak.get_copy should raise if it encounters a Custom_tag?

Shallow copies aren't even type-safe in the proper sense of the word,
since copying a value of abstract type breaks abstraction. This is why
Obj.dup is in the Obj module. Really Weak.get_copy shouldn't exist, or
should at least be part of Obj not Weak.

I suspect Weak.get_copy only really exists for implementing the weak
hash tables, and even there using it means that you could give Weak.Make
a valid argument whose equal function used `==` and it would give
incorrect results.

Regards,

Leo

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

["Martin R. Neuhäußer"]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

Just for the sake of completeness:
The test case works correctly, if ephemerons are used to simulate a weak set. See the corresponding test case „gcbug3.ml“.
So there seems to at least exist a (sub-optimal) work-around in OCaml 4.03.0 and later.

Best,
Martin

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 204 bytes --]

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[François Bobot]

On 18/06/2016 19:39, "Martin R. Neuhäußer" wrote:
> Just for the sake of completeness:
> The test case works correctly, if ephemerons are used to simulate a weak set. See the corresponding test case „gcbug3.ml“.
> So there seems to at least exist a (sub-optimal) work-around in OCaml 4.03.0 and later.
>

Thank you for all these repro cases and investigations!

> I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail.

You should not hesitate since you created nice repro cases that behave differently on ocaml 
versions. Perhaps at the end it is not a bug, but it is easier to keep track of problems on a bug 
tracker. In the mean time I will look at it.


Thanks,


-- 
François

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[François Bobot]

On 18/06/2016 18:54, Leo White wrote:
>> Shallow copies don't seem very safe in the presence of out-of-OCaml-heap C pointers [*].
>> Perhaps Weak.get_copy should raise if it encounters a Custom_tag?
>
> Shallow copies aren't even type-safe in the proper sense of the word,
> since copying a value of abstract type breaks abstraction. This is why
> Obj.dup is in the Obj module. Really Weak.get_copy shouldn't exist, or
> should at least be part of Obj not Weak.

That's interesting, perhaps I should not have given access to get_copy in Ephemeron.

>
> I suspect Weak.get_copy only really exists for implementing the weak
> hash tables, and even there using it means that you could give Weak.Make
> a valid argument whose equal function used `==` and it would give
> incorrect results.

The weak hashtable of ephemerons doesn't use get_copy, specifically because of that. The weak set of 
Weak are mostly used for hashconsing where the equal function given to Make must not be (==). But 
for the other use-case, it could be indeed a problem.


However it would be interesting to keep the nice property that `get_copy` give us for the 
implementation of weak structure, without using unsafe features. The nice property is that a value 
is not kept alive just because it collides with a value that is often accessed. Perhaps a kind of 
transient roots, which are considered as root only at the end of the each part of mark phase, would 
allow to mark the value as few time as possible. But the API would not be pretty...

-- 
François

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Alain Frisch]

On 21/06/2016 14:43, François Bobot wrote:
> However it would be interesting to keep the nice property that
> `get_copy` give us for the implementation of weak structure, without
> using unsafe features. The nice property is that a value is not kept
> alive just because it collides with a value that is often accessed.
> Perhaps a kind of transient roots, which are considered as root only at
> the end of the each part of mark phase, would allow to mark the value as
> few time as possible. But the API would not be pretty...

Not directly related to this thread, but it might be worth mentioning 
that for our use case (hash-consing terms), disabling the shallow copy 
(i.e. using get instead of get_copy) lead to very significant 
performance gains.  The main explanation is probably that it reduces 
quite a bit the pressure on the GC (more than the avoided cost of the 
copy itself, I guess).  Another possible explanation is that keeping 
some values a bit longer in the weak set is actually a good thing in our 
usage scenario (we keep some derived values, computed only when the term 
is first inserted in the weak set); this is also exacerbated by the 
lower rate of the GC (the first explanation).  We are "saved" by the 
fact that phases that exercises the weak set a lot are alternating with 
other GC-intensive phases (which guarantee that the weak set are 
eventually pruned).

Perhaps the "official" Weak.Make should at least allow to use get 
instead of get_copy?


Alain

Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[François Bobot]

On 21/06/2016 21:37, Alain Frisch wrote:
>
> Not directly related to this thread, but it might be worth mentioning
> that for our use case (hash-consing terms), disabling the shallow copy
> (i.e. using get instead of get_copy) lead to very significant
> performance gains.

It's an interesting report. The contrary of Doligez and Cuoq paper[1], but the usage of hashconsing 
in Frama-C is quite different from yours.

>
> Perhaps the "official" Weak.Make should at least allow to use get
> instead of get_copy?

Before complicating the API, I would prefer to check what happend if we get ride of get_copy but 
keep the nice property (by complicating the runtime...). In your case forcing the keeping of some 
value explicitly with a usual "hard" cache (with LRU stategy?) on top of the weak set could be 
sufficient and more intelligible.

[1]:  http://cristal.inria.fr/~doligez/publications/cuoq-doligez-mlw-2008.pdf

-- 
François

AW: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

[Neuhaeusser, Martin]

I just opened a Mantis ticket for the issue: http://caml.inria.fr/mantis/view.php?id=7279


-----Ursprüngliche Nachricht-----
Von: caml-list-request@inria.fr [mailto:caml-list-request@inria.fr] Im Auftrag von François Bobot
Gesendet: Dienstag, 21. Juni 2016 13:56
An: caml-list@inria.fr
Betreff: Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module...

On 18/06/2016 19:39, "Martin R. Neuhäußer" wrote:
> Just for the sake of completeness:
> The test case works correctly, if ephemerons are used to simulate a weak set. See the corresponding test case „gcbug3.ml“.
> So there seems to at least exist a (sub-optimal) work-around in OCaml 4.03.0 and later.
>

Thank you for all these repro cases and investigations!

> I hesitate a bit before opening an „official“ issue on Mantis as I might very well overlook some detail.

You should not hesitate since you created nice repro cases that behave differently on ocaml versions. Perhaps at the end it is not a bug, but it is easier to keep track of problems on a bug tracker. In the mean time I will look at it.


Thanks,


--
François

--
Caml-list mailing list.  Subscription management and archives:
https://sympa.inria.fr/sympa/arc/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs