understanding weak

understanding weak

[Warren Harris]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

I'd like to understand better how ocaml's weak pointers operate.  
First, although it doesn't seem to be specified in the documentation,  
I assume that weak pointers will *not* be reclaimed (e.g. from a weak  
hash table) if the program retains some other reference to the object.  
I.e. the weak pointer must be the last remaining pointer to the object  
for reclamation to occur.

My second question relates specifically to my application. I would  
like to have a primary cache of objects, and a secondary index into  
sub-objects referenced from the primary cache. I.e. CacheA references  
objects of type A; objects of type A reference objects of type B;  
CacheB references objects of type B. I would like to guarantee that  
weak references in CacheB are not flushed unless the corresponding  
reference from CacheA is first flushed. I assume will be the case if a  
non-weak reference from A to B is maintained. Can anyone verify?

Thanks,

Warren Harris
Metaweb Technologies


[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3739 bytes --]

Re: [Caml-list] understanding weak

[Martin Jambon]

Warren Harris wrote:
> I'd like to understand better how ocaml's weak pointers operate. First,
> although it doesn't seem to be specified in the documentation, I assume
> that weak pointers will *not* be reclaimed (e.g. from a weak hash table)
> if the program retains some other reference to the object. I.e. the weak
> pointer must be the last remaining pointer to the object for reclamation
> to occur.

Yes, otherwise the program would crash.

> My second question relates specifically to my application. I would like
> to have a primary cache of objects, and a secondary index into
> sub-objects referenced from the primary cache. I.e. CacheA references
> objects of type A; objects of type A reference objects of type B; CacheB
> references objects of type B. I would like to guarantee that weak
> references in CacheB are not flushed unless the corresponding reference
> from CacheA is first flushed. I assume will be the case if a non-weak
> reference from A to B is maintained. Can anyone verify?

Weak pointers are useful for data that should be shared by several
independent "users" and discarded when the number of users drops to zero.

I wouldn't call your CacheB a cache if it is a weak array or a weak hash
table. It is a table of shared objects. These objects of type B are
shared by objects of type A and possibly other users of CacheB.



Martin

-- 
http://mjambon.com/

Re: [Caml-list] understanding weak

[Daniel Bünzli]

Le 31 oct. 08 à 03:14, Martin Jambon a écrit :

> Warren Harris wrote:
>> I'd like to understand better how ocaml's weak pointers operate.  
>> First,
>> although it doesn't seem to be specified in the documentation, I  
>> assume
>> that weak pointers will *not* be reclaimed (e.g. from a weak hash  
>> table)
>> if the program retains some other reference to the object. I.e. the  
>> weak
>> pointer must be the last remaining pointer to the object for  
>> reclamation
>> to occur.
>
> Yes, otherwise the program would crash.

No since Weak.get returns an option type. As written the documentation  
sounds like the binding could disappear from the array even though the  
program still has references to the value. This could be done,  
wouldn't be usefull, but wouldn't crash the program.

Daniel

Re: [Caml-list] understanding weak

[Martin Jambon]

Daniel Bünzli wrote:
> 
> Le 31 oct. 08 à 03:14, Martin Jambon a écrit :
> 
>> Warren Harris wrote:
>>> I'd like to understand better how ocaml's weak pointers operate. First,
>>> although it doesn't seem to be specified in the documentation, I assume
>>> that weak pointers will *not* be reclaimed (e.g. from a weak hash table)
>>> if the program retains some other reference to the object. I.e. the weak
>>> pointer must be the last remaining pointer to the object for reclamation
>>> to occur.
>>
>> Yes, otherwise the program would crash.
> 
> No since Weak.get returns an option type. As written the documentation
> sounds like the binding could disappear from the array even though the
> program still has references to the value. This could be done, wouldn't
> be usefull, but wouldn't crash the program.

let x = (1, 2);;
let wa = Weak.create 10;;
Weak.set wa 0 (Some x);;
...
print_int (fst x);;

(fst x) would certainly cause funny effects if x were GC'ed at an
arbitrary time after it has been added to the weak array.

An object can be reclaimed by the GC only if there is no reference to
it. This remains true. Adding an object to a weak array just doesn't
count as a reference.



Martin

-- 
http://mjambon.com/

Re: [Caml-list] understanding weak

[Daniel Bünzli]

Le 31 oct. 08 à 15:57, Martin Jambon a écrit :

> (fst x) would certainly cause funny effects if x were GC'ed at an
> arbitrary time after it has been added to the weak array.

You are confusing reclaiming the weak pointer (the Some x) and  
reclaiming what the weak pointer refers to (the x). It's the former  
the OP was talking about.

Best,

Daniel

Re: [Caml-list] understanding weak

[Martin Jambon]

Daniel Bünzli wrote:
> 
> Le 31 oct. 08 à 15:57, Martin Jambon a écrit :
> 
>> (fst x) would certainly cause funny effects if x were GC'ed at an
>> arbitrary time after it has been added to the weak array.
> 
> You are confusing reclaiming the weak pointer (the Some x) and
> reclaiming what the weak pointer refers to (the x). It's the former the
> OP was talking about.

I don't see what makes you define (Some x) as a weak pointer.


Martin
-- 
http://mjambon.com/

Re: [Caml-list] understanding weak

[Aleksey Nogin]

On 31.10.2008 07:57, Martin Jambon wrote:

> let x = (1, 2);;
> let wa = Weak.create 10;;
> Weak.set wa 0 (Some x);;
> ...
> print_int (fst x);;
> 
> (fst x) would certainly cause funny effects if x were GC'ed at an
> arbitrary time after it has been added to the weak array.
> 
> An object can be reclaimed by the GC only if there is no reference to
> it. This remains true. Adding an object to a weak array just doesn't
> count as a reference.
> 
Martin,

You are answering the wrong question - you are answering "could x be
GCed too early?" - the answer is obviously "no". However, the initial
question was "could (Some x) be removed from wa too early by GC - before
x is orphaned?" The answer is "we'd hope not", but the documentation is
somewhat ambiguous.

Aleksey

Re: [Caml-list] understanding weak

[Martin Jambon]

Aleksey Nogin wrote:
> On 31.10.2008 07:57, Martin Jambon wrote:
> 
>> let x = (1, 2);;
>> let wa = Weak.create 10;;
>> Weak.set wa 0 (Some x);;
>> ...
>> print_int (fst x);;
>>
>> (fst x) would certainly cause funny effects if x were GC'ed at an
>> arbitrary time after it has been added to the weak array.
>>
>> An object can be reclaimed by the GC only if there is no reference to
>> it. This remains true. Adding an object to a weak array just doesn't
>> count as a reference.
>>
> Martin,
> 
> You are answering the wrong question - you are answering "could x be
> GCed too early?" - the answer is obviously "no". However, the initial
> question was "could (Some x) be removed from wa too early by GC - before
> x is orphaned?" The answer is "we'd hope not", but the documentation is
> somewhat ambiguous.

OK. I think now I understand the question :-)
And we all agree that the documentation could be clearer.

What I'm wondering is why there is one Weak.set function that takes an
option as argument rather than a pair Weak.set/Weak.clear.

What is certain is that some (re)boxing is done by Weak.get:

# let a = Weak.get wa 0;;
val a : (int * int) option = Some (1, 2)
# let b = Weak.get wa 0;;
val b : (int * int) option = Some (1, 2)
# a == b;;
- : bool = false

Of course it doesn't tell whether the original (Some x) is stored as-is
but copied by Weak.get or if x is stored unwrapped.

I think I'll just take a look at the implementation. That can only be a
good thing.


Martin
-- 
http://mjambon.com/

Re: [Caml-list] understanding weak

[Martin Jambon]

Aleksey Nogin wrote:
> On 31.10.2008 07:57, Martin Jambon wrote:
> 
>> let x = (1, 2);;
>> let wa = Weak.create 10;;
>> Weak.set wa 0 (Some x);;
>> ...
>> print_int (fst x);;
>>
>> (fst x) would certainly cause funny effects if x were GC'ed at an
>> arbitrary time after it has been added to the weak array.
>>
>> An object can be reclaimed by the GC only if there is no reference to
>> it. This remains true. Adding an object to a weak array just doesn't
>> count as a reference.
>>
> Martin,
> 
> You are answering the wrong question - you are answering "could x be
> GCed too early?" - the answer is obviously "no". However, the initial
> question was "could (Some x) be removed from wa too early by GC - before
> x is orphaned?" The answer is "we'd hope not", but the documentation is
> somewhat ambiguous.

So I checked the implementation and it turns out that x is unboxed from
(Some x) before being added to the weak array.

x is really the value that matters and of course not keeping any
reference to (Some x) has no importance.

The weak pointer is x (or a null value), stored as a cell of the weak array.


Martin
-- 
http://mjambon.com/

Re: understanding weak

[Rémi Vanicat]

Alain Frisch <alain@frisch.fr> writes:

> Warren Harris wrote:
>> On Oct 30, 2008, at 11:48 AM, CUOQ Pascal - Pascal.CUOQ@cea.fr wrote:
>>> In short: don't use weak pointers to make caches.
>>
>> Thanks for the advice -- but I thought this was exactly what weak
>> hash tables were intended for.
>
> Although there is some similarity between a weak table and a cache
> ("store values, but feel free to get rid of them"), they are really
> different beasts: a good implementation of a weak table should release
> memory as soon as possible to avoid memory leak; a good implementation
> of a cache should instead keep data as long as possible (until there
> is a good reason to release it, like resource exhaustion).

It's true, but it could be interesting to have a cache pointer/cache
table provided with the garbage collector: it's the GC that know when
there is not enough memory left, and that it should free those cache
pointer to have some.

-- 
Rémi Vanicat

RE: understanding weak

[CUOQ Pascal]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]


Warren Harris <warren@metaweb.com> wrote:
> I'd like to understand better how ocaml's weak pointers operate. 

You will be interested in the following important article:
http://portal.acm.org/citation.cfm?id=1411308
:)
 
>First, although it doesn't seem to be specified in the documentation,  
>I assume that weak pointers will *not* be reclaimed (e.g. from a weak  
>hash table) if the program retains some other reference to the object.  

Exactly.

>My second question relates specifically to my application. I would  
>like to have a primary cache of objects, and a secondary index into  
>sub-objects referenced from the primary cache. I.e. CacheA references  
>objects of type A; objects of type A reference objects of type B;  
>CacheB references objects of type B. I would like to guarantee that  
>weak references in CacheB are not flushed unless the corresponding  
>reference from CacheA is first flushed. I assume will be the case if a  
>non-weak reference from A to B is maintained.

The non-weak reference from A to B prevents B being unreachable
without A being unreachable, so yes, a reference from CacheB can
not disappear without the reference from CacheA disappearing
earlier or simultaneously.
This said, if what you want is really a cache, you would probably be
better off fixing its size and renewal strategy yourself than letting
the GC do it for you (by using weak pointers). What it will do has almost
no chance of being the best compromise between memory use and
speed for your application, and it may change without notice from
one version to the other. In short: don't use weak pointers to make caches.

>Can anyone verify?

If you want to experiment to confirm your impressions, the function
Gc.full_major is your friend.

Pascal





[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 3459 bytes --]