[Caml-list] Memory leak

[Caml-list] Memory leak

[Martin Jambon]

Hello,

Is it possible that non-negligible amounts of unreachable data are never
freed by the GC?

My program seems to waste memory, even with Gc.max_overhead set to 50 (or
even 0).
What should I do?

The program is written only in OCaml.
It has been tested using version 3.04 and last friday's working version
(3.04+9 (2002-04-04)).

No dangerous feature of the language is used except Marshal, I think.
The program may sequentially handle several gigabytes of marshaled data,
but doesn't crash and always produces correct results.

Example of Gc.stat (OCaml 3.04):

minor_words: 1746668652
promoted_words: 104792423
major_words: 232260045
minor_collections: 121
major_collections: 75727
heap_words: 250683392 (* bytes *)
heap_chunks: 941
live_words: 28221788
live_blocks: 4657360
free_words: 34443371
free_blocks: 772158
largest_free: 21914
fragments: 5689
compactions: 11

In this case, the size of the heap should stay below 10 MB. Here, it is
250 MB.
Removal of some parts of the program doesn't produce expected results
[...].


Comments in module Gc say [about Gc.finalise]:
   Some constant values can be heap-allocated but never deallocated
   during the lifetime of the program, for example a list of integer
   constants; this is also implementation-dependent.

How do I know if my program uses such permanent data?
Should I use Gc.finalise to look for missing deallocations?


Hoping I will get some help...
Thanks in advance,


Martin

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Memory leak

[Brian Rogoff]

On Mon, 15 Apr 2002, Martin Jambon wrote:

> Hello,
>
> Is it possible that non-negligible amounts of unreachable data are never
> freed by the GC?

Sure. There was a great demo of this on the translator's list for the
O'Reilly book; create a representation of stacks based on arrays with a
top of stack index. Popping items off of the stack by moving the TOS
pointer around won't free them. Could you be doing something like that?

-- Brian
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Memory leak

[Scott Cyphers]

> Sure. There was a great demo of this on the translator's list for the
> O'Reilly book; create a representation of stacks based on arrays with a
> top of stack index. Popping items off of the stack by moving the TOS
> pointer around won't free them. Could you be doing something like that?

In that particular case, the stack implementation is a lot uglier when you
need to provide the ability to neutralize array elements of arbitrary type
as you pop the stack.  Sometimes you get lulled into thinking the GC works
completely invisibly.

Lisp Machines had special versions of GC that would first run through a list
of GC user-definable initializers that would clear off history lists and
other places that tended to accumulate garbage.  With each new release, new
garbage gatherers and problems like the stack implementation above needed to
be tracked down and adjusted or added to the list.  You get so you notice
that kind of code and automatically unhook pointers when you're done with
them.

I think I caused the biggest leak when I added a compiler optimizer that
removed reads for values that were never used.  One part of this special
garbage collection process played some GC space games and required a memory
read to start the garbage collector, so we had to introduce read for effect.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Memory leak

[Markus Mottl]

On Tue, 16 Apr 2002, Scott Cyphers wrote:
> > Sure. There was a great demo of this on the translator's list for the
> > O'Reilly book; create a representation of stacks based on arrays with a
> > top of stack index. Popping items off of the stack by moving the TOS
> > pointer around won't free them. Could you be doing something like that?
> 
> In that particular case, the stack implementation is a lot uglier when you
> need to provide the ability to neutralize array elements of arbitrary type
> as you pop the stack.  Sometimes you get lulled into thinking the GC works
> completely invisibly.

In case you ever need such functionality (stacks based on arrays), you
might want to try the library of resizable arrays called RES, which also
handles resizable strings:

  http://www.ai.univie.ac.at/~markus/home/ocaml_sources.html

It plays some naughty low-level tricks to free memory of unreachable
values, which occur when some resizable array shrinks.

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Memory leak

[Brian Rogoff]

On Wed, 17 Apr 2002, Markus Mottl wrote:
> On Tue, 16 Apr 2002, Scott Cyphers wrote:
> > > Sure. There was a great demo of this on the translator's list for the
> > > O'Reilly book; create a representation of stacks based on arrays with a
> > > top of stack index. Popping items off of the stack by moving the TOS
> > > pointer around won't free them. Could you be doing something like that?
> >
> > In that particular case, the stack implementation is a lot uglier when you
> > need to provide the ability to neutralize array elements of arbitrary type
> > as you pop the stack.  Sometimes you get lulled into thinking the GC works
> > completely invisibly.
>
> In case you ever need such functionality (stacks based on arrays), you
> might want to try the library of resizable arrays called RES, which also
> handles resizable strings:
>
>   http://www.ai.univie.ac.at/~markus/home/ocaml_sources.html

Hey, don't they say "Never use Obj.magic!"? ;-)

I thought the issue that Scott caught on the translator's list was nice,
and very easy to overlook. The safe way to do this without using magic is
to either change the interface to provide a value of the element type or
to use options underneath and add another level of packing and unpacking.

I don't know if that issue is FAQ worthy, but it belongs in some kind of
"gotcha" list for Caml programmers. Really anyone using a GCed language
with arrays may see that one...

-- Brian
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Memory leak

[Martin Jambon]

On Mon, 15 Apr 2002, Martin Jambon wrote:

> Hello,
>
> Is it possible that non-negligible amounts of unreachable data are never
> freed by the GC?
>
> My program seems to waste memory, even with Gc.max_overhead set to 50 (or
> even 0).
> What should I do?

It appears finally that explicit calls to Gc.compact solve the problem (or
at least one of my problems).

Setting Gc.max_overhead to a given value (e.g. 50%) didn't work in this
case.
Symptom: compactions are performed until a certain point
(in my case, 67 compactions), and then no compaction is triggered anymore
for a very long (infinite?) time, resulting in a linear increase of the
memory.



Thanks to all those that sent me their suggestions.

Au passage, un grand merci à l'équipe Caml pour son travail de très haute
qualité.


Martin Jambon

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners