[Caml-list] Memory usage of ocaml program

[Caml-list] Memory usage of ocaml program

[Chengqi Song]

[-- Attachment #1: Type: text/plain, Size: 600 bytes --]

I have the same program (same binary) running in different modes. From
GC stats I see they're using the same amount of heap now (209m) and
the stack is small. But in top's RES column, I see one is using 270m
memory and the other one is using 622m. Forcing heap compaction does
not change the numbers.

I don't quite understand this difference. A program's memory usage
should be code, static data, heap and stack. For the same binary, code
and static data segment should be the same, and now that GC stats
tells me heap and stack are the same, why there is a 352m difference
in memory usage?

Thanks

[-- Attachment #2: Type: text/html, Size: 658 bytes --]

Re: [Caml-list] Memory usage of ocaml program

[Gabriel Scherer]

If you use the C bindings (or a library that does), you may have
allocated memory from C, outside the OCaml heap. This memory is not
handled by the GC and will not appear in the GC stats.

On Fri, Nov 23, 2012 at 3:34 AM, Chengqi Song <songcq@gmail.com> wrote:
> I have the same program (same binary) running in different modes. From
> GC stats I see they're using the same amount of heap now (209m) and
> the stack is small. But in top's RES column, I see one is using 270m
> memory and the other one is using 622m. Forcing heap compaction does
> not change the numbers.
>
> I don't quite understand this difference. A program's memory usage
> should be code, static data, heap and stack. For the same binary, code
> and static data segment should be the same, and now that GC stats
> tells me heap and stack are the same, why there is a 352m difference
> in memory usage?
>
> Thanks

Re: [Caml-list] Memory usage of ocaml program

[Edgar Friendly]

Also Bigarrays, although this falls under "libraries that use C bindings".

E.

On 11/23/2012 1:40 AM, Gabriel Scherer wrote:
> If you use the C bindings (or a library that does), you may have
> allocated memory from C, outside the OCaml heap. This memory is not
> handled by the GC and will not appear in the GC stats.
>
> On Fri, Nov 23, 2012 at 3:34 AM, Chengqi Song <songcq@gmail.com> wrote:
>> I have the same program (same binary) running in different modes. From
>> GC stats I see they're using the same amount of heap now (209m) and
>> the stack is small. But in top's RES column, I see one is using 270m
>> memory and the other one is using 622m. Forcing heap compaction does
>> not change the numbers.
>>
>> I don't quite understand this difference. A program's memory usage
>> should be code, static data, heap and stack. For the same binary, code
>> and static data segment should be the same, and now that GC stats
>> tells me heap and stack are the same, why there is a 352m difference
>> in memory usage?
>>
>> Thanks

Re: [Caml-list] Memory usage of ocaml program

[Gerd Stolpmann]

Am Freitag, den 23.11.2012, 10:34 +0800 schrieb Chengqi Song:
> I have the same program (same binary) running in different modes. From
> GC stats I see they're using the same amount of heap now (209m) and
> the stack is small. But in top's RES column, I see one is using 270m
> memory and the other one is using 622m. Forcing heap compaction does
> not change the numbers.
> 
> I don't quite understand this difference. A program's memory usage
> should be code, static data, heap and stack. For the same binary, code
> and static data segment should be the same, and now that GC stats
> tells me heap and stack are the same, why there is a 352m difference
> in memory usage?

Here are some ideas:

 - First, RES or RSS is not identical to the memory the program
   has allocated. It is the memory that is currently used in RAM
   (resident memory). This does not count memory swapped out,
   memory-mapped files, and memory that was allocated but never written
   to.

   For a better impression what is happening look at the memory
   map of the running process: On Linux this is /proc/$pid/maps, and on
   OS X you can get it with the vmmap command.

 - Sometimes it is not possible to give memory back to the
   kernel. Normally, a program gets memory from the kernel
   in big chunks, and it can only give it back when such a
   chunk becomes empty, or when the chunk can be shrunk (i.e. memory
   at the end of the chunk becomes empty). The details of this
   are very OS-dependent. OCaml calls malloc/free, and does it very
   best to free what is unused, but libc finally decides about the real 
   allocation chunks.

   On traditional Unix there was even only one such chunk (called
   "the heap"), and you could only extend or shrink it. When memory
   became free in the middle of this chunk it could never be really
   released.

   On modern OS this is normally better, though, e.g. Linux libc
   can manage several chunks. But nevertheless you can still
   run into such issues.

Gerd

> Thanks

-- 
------------------------------------------------------------
Gerd Stolpmann, Darmstadt, Germany    gerd@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details:        http://www.camlcity.org/contact.html
Company homepage:       http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.
------------------------------------------------------------