Memory usage

Memory usage

[Dorlet Emmanuel]

Hello,

I am in charge of a very very simple test on the ability of Ocaml
garbage to free unused memory.
Obvioulsy, Ocaml does the work properly (but not in the case of the
global context, where every variable allocation is consider as a
living reference, even if that is not true).
But I can't see, from my (perhaps too fast) reading of the documentation,
how to compute a global memory usage and how I can monitor it.
Does any one has a good and simple suggestion?


__________________________________________________________

  Emmanuel DORLET
  Commissariat à l'Energie Atomique
  Centre de Saclay - DRN-DMT/SYSCO/LGLS
  Bâtiment 460 - Pièce 19 A
  Téléphone : 01 69 08 91 17
  Télécopie : 01 69 08 96 96
  e-mail: Emmanuel.Dorlet@cea.fr
__________________________________________________________

Re: Memory usage

[David Mentré]

"Dorlet Emmanuel" <Emmanuel.Dorlet@cea.fr> writes:

> Does any one has a good and simple suggestion?

To monitor global memory usage, I use the following code :

# let stat = Gc.stat ()
  and control = Gc.get () in
  let max_words_total = stat.Gc.heap_words + control.Gc.minor_heap_size in
  Printf.printf "Maximum memory used (allocated): %d kBytes\n" 
          (max_words_total * Sys.word_size / 8 / 1024);;

  Maximum memory used (allocated): 624 kBytes
- : unit = ()

It gives you the total heap allocated.

Moreover, at the following URL, you'll find a program, "size", to
compute memory size of an OCaml value :

 http://www.lri.fr/~filliatr/software.en.html


If you have more precise ways to know memory behavior, I would be very
interested in them.

Best regards,
d.
-- 
 David.Mentre@irisa.fr -- http://www.irisa.fr/prive/dmentre/
 Opinions expressed here are only mine.

Re: memory usage

[Sylvain Le Gall]

Can you make sure that all your function terminate and are joined
(Thread.join). I think this will help to make sure that the thread exit
and call thread_kill (see OCaml source code).

If you take a look at thread_kill there is a function stat_free + set to
NULL things called stack_low, stack_high... Maybe all the data you are
seeing come from this...

I am not sure that Thread.join will free anything, but it will help you
to be sure that your thread has exited correctly.

Regards,
Sylvain Le Gall

Re: memory usage

[Sylvain Le Gall]

On 12-01-2009, John Lepikhin <john@ispsystem.com> wrote:
>
>> > Each thread is killed after work is done
>> How do you "kill" the threads? I hope this is just a figure of
>> speech for "I do an orderly shutdown of each thread after work is done."
>
> Well, that was consequence of my bad English :-) Threads finish their
> work and exit. I also made a simple wrapper to Thread.create to be sure
> that all work inside threads is done:
>
> module MyThread =
>   let create f p =
>     let dowork _ =
>       (* log thread creation *)
>       f p;
>       (* log thread shutdown *)
>     in
>     Thread.create dowork ()
> end

Do you use some kind of Thread.join ? 

Regards
Sylvain Le Gall

Re: memory usage

[Sylvain Le Gall]

On 12-01-2009, John Lepikhin <john@ispsystem.com> wrote:
>> Starting point should be to call this periodically:
>> 
>>       Gc.compact ();
>>       let stat = Gc.stat () in
>>       let live_words = stat.Gc.live_words in
>>       eprintf "live words %d\n%!" live_words;
>> 
>> which will tell you how many words (ie 4 or 8 byte chunks) are
>> reachable according to the garbage collector.
>
> Richard, here is result (statistics was saved every 10 seconds):
>
> live words - RSS:
>
> 186980 - 12380KB <-- after first 10 seconds of work
> 154156 - 18232KB
> 153923 - 19648KB
> ...
> after 10 minutes of work:
> 203842 - 33436KB
> 170559 - 33528KB
> 187018 - 33664KB
> 71626 - 33592KB
>
> Sometimes live words drops down to 40.000. But RSS always stay near
> 30-50MB.
>

To get real memory used, (Sys.word_size * live_word / 8). Do you use
out-of-heap datastructure that can use memory ? (malloc-ed
datastructure).

Regards,
Sylvain Le Gall

memory usage

[John Lepikhin]

Hello,

I have developed an application and have issue with memory usage.
Application was designed for work as server, 24x7. After several hours
of work, RSS memory grows up to 40-50MB. Each thread must not use more
than 100-200KB of memory (maximum 20 threads are run at the same time).
GC debug information:

Growing heap to 3840k bytes
Growing heap to 4320k bytes
Shrinking heap to 3840k bytes
Growing heap to 4320k bytes
Growing heap to 4800k bytes
Growing heap to 5280k bytes
Shrinking heap to 4800k bytes
Shrinking heap to 4320k bytes
Shrinking heap to 3840k bytes
Growing heap to 4320k bytes
Growing heap to 4800k bytes
Growing heap to 5280k bytes
Shrinking heap to 4800k bytes
Shrinking heap to 4320k bytes
Shrinking heap to 3840k bytes
Shrinking heap to 3360k bytes
Growing heap to 3840k bytes
Growing heap to 4320k bytes
Growing heap to 4800k bytes
Growing heap to 960k bytes
Shrinking heap to 4320k bytes
Shrinking heap to 3840k bytes
Shrinking heap to 3360k bytes
Shrinking heap to 2880k bytes
Growing heap to 3360k bytes
Shrinking heap to 2880k bytes
Shrinking heap to 2400k bytes
Growing heap to 2880k bytes
Growing heap to 3360k bytes
Growing heap to 3840k bytes
Shrinking heap to 3360k bytes
Shrinking heap to 2880k bytes
Growing heap to 3360k bytes
Shrinking heap to 2880k bytes

As you can see, heap size never gets bigger 5-6MB. I made core dump of
process. 90% of memory was filled with values which are created inside
threads and never been copied outside of them. Each thread is killed
after work is done, I am absolutely sure in it; all unused file
descriptors are closed. Playing with Gc.set has not brought notable
results.

Please, give me some start point to find the roots of the problem.

Sorry for my English.

memory usage

[Jean Krivine]

Dear list members,

I am trying to run a stochastic simulator (written in ocaml) on a huge
data set and I have the following error message:

sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Fatal error: out of memory.

My system:

Mac Pro running OS X 10.5.4
Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
Memory:  10 GB 800 MHz DDR2 FB-DIMM

Does someone know what happened? Do you have any idea of any parameter
I could tune in order to avoid that?

Thank you very much!

Jean