[Caml-list] Garbage collector and memory fragmentation

[Caml-list] Garbage collector and memory fragmentation

[francois bereux]

<This question is not specific to OCaml, but rather to garbage
collection.>

I have a program (in Fortran90) that uses many (> 100.000) small lists
of integers ( about 100 elements per list ). Each list is built
incrementally => I allocate many small blocks. It appears that this
leads to memory fragmentation. For instance, even though I deallocate
all the lists, the memory used by the program does not seem to really
decrease.
My question is : does a garbage collector (for instance the one in
OCaml) deal with this kind of issues ( defragmentation of the memory )
in a situation similar to mine : many small lists of elements ?

I apologize if this question is too much off-topic from the list.

--
****************************************************
*   François BÉREUX
*
*   Departement Antennes & Radomes
*   THALES Systemes Aeroportes
*   Centre René Mouchotte
*   La Clef de Saint-Pierre
*   1, Boulevard Jean Moulin
*   F-78852 Élancourt Cedex
*   FRANCE
*
*   tél : +33-1-34.59.73.12
*   fax : +33-1-34.59.70.54
*   mél : francois.bereux@fr.thalesgroup.com
****************************************************


-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Garbage collector and memory fragmentation

[Markus Mottl]

On Fri, 24 May 2002, francois bereux wrote:
> My question is : does a garbage collector (for instance the one in
> OCaml) deal with this kind of issues ( defragmentation of the memory )
> in a situation similar to mine : many small lists of elements ?

The GC employed by OCaml is able to compact the heap, which means
it essentially defragments memory. You can parameterize it with the
frequency of such compactions (they are very expensive), turn them off
completely or force them manually. Take a look at the interface of the
Gc-module in the distribution for more details.

Regards,
Markus Mottl

-- 
Markus Mottl                                             markus@oefai.at
Austrian Research Institute
for Artificial Intelligence                  http://www.oefai.at/~markus
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Garbage collector and memory fragmentation

[Martin Jambon]

At 12:22 24/05/02 +0200, Markus Mottl wrote:
>On Fri, 24 May 2002, francois bereux wrote:
> > My question is : does a garbage collector (for instance the one in
> > OCaml) deal with this kind of issues ( defragmentation of the memory )
> > in a situation similar to mine : many small lists of elements ?
>
>The GC employed by OCaml is able to compact the heap, which means
>it essentially defragments memory. You can parameterize it with the
>frequency of such compactions (they are very expensive), turn them off
>completely or force them manually. Take a look at the interface of the
>Gc-module in the distribution for more details.

Il y a 2 options :
  - compaction automatique dès que l'espace de mémoire inutilisé dépasse x 
% (en utilisant Gc.set et en spécifiant le champ max_overhead : Gc.set { 
(Gc.get ()) with Gc.max_overhead = 200 };;)
  - compaction explicite par appel à Gc.compact de temps en temps.

La première solution est la plus élégante, mais il y a un bug (cf. bug 
report numero 1111) qui apparaît dans certains cas de figure non 
déterminés. Je tenais juste à vous l'indiquer au cas où vous tomberiez 
dessus...

Martin

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Garbage collector and memory fragmentation

[Xavier Leroy]

> <This question is not specific to OCaml, but rather to garbage
> collection.>
> 
> I have a program (in Fortran90) that uses many (> 100.000) small lists
> of integers ( about 100 elements per list ). Each list is built
> incrementally => I allocate many small blocks. It appears that this
> leads to memory fragmentation. For instance, even though I deallocate
> all the lists, the memory used by the program does not seem to really
> decrease.
> My question is : does a garbage collector (for instance the one in
> OCaml) deal with this kind of issues ( defragmentation of the memory )
> in a situation similar to mine : many small lists of elements ?

It depends on the allocation and garbage collection algorithms used.
For instance, relocating garbage collectors (stop&copy collectors, or
in-place compactors) prevent fragmentation, while garbage collectors
that do not move objects around (mark&sweep, reference count) are
vulnerable to fragmentation issues.  Even for non-relocating
collectors, the allocation strategy also has an impact on how much
fragmentation occurs.  

To learn more, I'd recommend Paul Wilson's excellent surveys:

ftp://ftp.cs.utexas.edu/pub/garbage/gcsurvey.ps    (on GC)
ftp://ftp.cs.utexas.edu/pub/garbage/allocsrv.ps    (on allocation)

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Garbage collector and memory fragmentation

[Christophe Raffalli]

Le lun 27/05/2002 à 10:19, Xavier Leroy a écrit :
> > <This question is not specific to OCaml, but rather to garbage
> > collection.>
> > 

> > My question is : does a garbage collector (for instance the one in
> > OCaml) deal with this kind of issues ( defragmentation of the memory )
> > in a situation similar to mine : many small lists of elements ?


Another point: the standard allocation (malloc) works in general with
pages: a page will really be freed if nothing in the page is used and
very often (always ?) malloc is bad at reusing pieces of a page that
have been deallocated.

Laguage with GC always deals with that issue (never perfectly, except
for stop&copy or other copacting GC) ! Otherwise, they will fail exactly
on the situation you describe which are very common in functionnal
programs.

-- 
Christophe Raffalli
Université de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tél: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] Garbage collector and memory fragmentation

[francois bereux]

Thanks to all who gave me answers. I have now a much clearer idea of the topic.
Sincerely yours,

François Béreux

francois bereux a écrit :

> <This question is not specific to OCaml, but rather to garbage
> collection.>
>
> I have a program (in Fortran90) that uses many (> 100.000) small lists
> of integers ( about 100 elements per list ). Each list is built
> incrementally => I allocate many small blocks. It appears that this
> leads to memory fragmentation. For instance, even though I deallocate
> all the lists, the memory used by the program does not seem to really
> decrease.
> My question is : does a garbage collector (for instance the one in
> OCaml) deal with this kind of issues ( defragmentation of the memory )
> in a situation similar to mine : many small lists of elements ?
>
> I apologize if this question is too much off-topic from the list.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners