[Caml-list] Memory fragmentation

[Caml-list] Memory fragmentation

[SerP]

[-- Attachment #1: Type: text/plain, Size: 1221 bytes --]

Hi!
We have developped a daemon on ocaml using the lwt lib(libev). It processes
about 800 requests per second, but it increases to 2 Gb in memory for a
hour of work. We have studied the problem for a long time, using mtrace,
mallinfo and other tools, and we tried to change GC params. We found out
that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will
make the daemon grow slower. mallinfо shows that the memory (1,5G) is
allocated using mmap (blkhd field in mallinfo struct) and arena size -
about 20M In this case, first calls of GC.compact compress the daemon to
200 Mb (live -110Mb) , and mallinfo show decreasing of total size of memory
allocated by mmap. The daemon keeps growing, but it allocates the memory to
arena (using brk), and calls of GC.compact leads to decrease of heap_bytes
to 200M, but arena size (1.5 Gb) does not decrease, just doing less
uordblks and more fordblks (fileds I.e., after first calls of GC.compact,
the daemon starts using brk much more than mmap, and cant memory given back
to the system.

If Gc.allocation_policy is first fit, memory usage is stable, and it's grow
very little, but worked very slowly.

Any suggestions are very welcome. Thanks

[-- Attachment #2: Type: text/html, Size: 1235 bytes --]

[Caml-list] Re: Memory fragmentation

[SerP]

[-- Attachment #1: Type: text/plain, Size: 1352 bytes --]

Ocaml 3.12.0., Lwt 2.3.1
On Apr 20, 2012 7:12 PM, "SerP" <serp256@gmail.com> wrote:

> Hi!
> We have developped a daemon on ocaml using the lwt lib(libev). It
> processes about 800 requests per second, but it increases to 2 Gb in memory
> for a hour of work. We have studied the problem for a long time, using
> mtrace, mallinfo and other tools, and we tried to change GC params. We
> found out that setting up GC.minor_heap_size=10Mb а
> major_heap_increment=2Мb will make the daemon grow slower. mallinfо shows
> that the memory (1,5G) is allocated using mmap (blkhd field in mallinfo
> struct) and arena size - about 20M In this case, first calls of GC.compact
> compress the daemon to 200 Mb (live -110Mb) , and mallinfo show decreasing
> of total size of memory allocated by mmap. The daemon keeps growing, but it
> allocates the memory to arena (using brk), and calls of GC.compact leads to
> decrease of heap_bytes to 200M, but arena size (1.5 Gb) does not decrease,
> just doing less uordblks and more fordblks (fileds I.e., after first calls
> of GC.compact, the daemon starts using brk much more than mmap, and cant
> memory given back to the system.
>
> If Gc.allocation_policy is first fit, memory usage is stable, and it's
> grow very little, but worked very slowly.
>
> Any suggestions are very welcome. Thanks
>

[-- Attachment #2: Type: text/html, Size: 1576 bytes --]

Re: [Caml-list] Memory fragmentation

[Török Edwin]

On 04/20/2012 06:12 PM, SerP wrote:
> Hi!
> We have developped a daemon on ocaml using the lwt lib(libev). It processes about 800 requests per second, but it increases to 2 Gb in memory for a hour of work. We have studied the problem for a long
> time, using mtrace, mallinfo and other tools,
> and we tried to change GC params. We found out that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will make the daemon grow slower.
> mallinfо shows that the memory (1,5G) is allocated using mmap (blkhd field in mallinfo struct) and arena size - about 20M In this case, first calls of GC.compact compress the daemon to 200 Mb (live
> -110Mb) , and mallinfo show decreasing of total size of memory allocated by mmap. The daemon keeps growing, but it allocates the memory to arena (using brk), and calls of GC.compact leads to decrease
> of heap_bytes to 200M, but arena size (1.5 Gb) does not decrease, just doing less uordblks and more fordblks (fileds I.e., after first calls of GC.compact, the daemon starts using brk much more than
> mmap, and cant memory given back to the system.

Sounds like a fragmentation problem in glibc's heap that causes increased VIRT usage.
IIRC glibc has a default threshold of 128k for deciding to use brk() vs mmap(), and OCaml's default major_heap_increment (124k) is just below that,
which would explain the behaviour you see, and also explain why using a larger major_heap_increment (2M) improves the situation:
 brk() cannot free "holes", while mmap arenas can.

You can use malloc_stats() to check whether OCaml gives back all memory to glibc or not, and you can try to use
an alternative malloc implementation (like jemalloc) that might cope better with fragmentation.

However this doesn't explain why memory keeps growing after a Gc.compact, unless calls to malloc
are mixed between the OCaml runtime, Lwt and libev. If that is the case then maybe it would help
to try and allocate memory in the OCaml's runtime by using mmap() directly.

Is your application completely single-threaded (since you're using lwt), or do you also use multiple threads?
I think that glibc caches mmap-ed areas even if you free them (i.e. they're still mapped into your process
but they might be changed via mprotect() to PROT_NONE), which is especially a problem if you use multiple threads.

Best regards,
--Edwin

Re: [Caml-list] Memory fragmentation

[SerP]

[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]

>
> If that is the case then maybe it would help to try and allocate memory in
> the OCaml's runtime by using mmap() directly.
>
Yes, i need to try this.

>
> Is your application completely single-threaded (since you're using lwt),
> or do you also use multiple threads?
> I think that glibc caches mmap-ed areas even if you free them (i.e.
> they're still mapped into your process but they might be changed via
> mprotect() to PROT_NONE), which is especially a problem if you use multiple
> threads.
>
Lwt use threads for some bloking system calls, so my application has
multiple threads.

Strange behaviour is that the before (first|second) compaction all ocaml
memory (1GB) was in mmaped area, but after all ocaml memory in placed in
brked memory and later all allocations occurs throught brk. major_heap - is
2MB


> Best regards,
> --Edwin
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 1838 bytes --]

Re: [Caml-list] Memory fragmentation

[ygrek]

On Fri, 20 Apr 2012 19:12:12 +0400
SerP <serp256@gmail.com> wrote:

> Hi!
> We have developped a daemon on ocaml using the lwt lib(libev). It processes
> about 800 requests per second, but it increases to 2 Gb in memory for a
> hour of work. We have studied the problem for a long time, using mtrace,
> mallinfo and other tools, and we tried to change GC params. We found out
> that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will
> make the daemon grow slower. mallinfо shows that the memory (1,5G) is
> allocated using mmap (blkhd field in mallinfo struct) and arena size -
> about 20M In this case, first calls of GC.compact compress the daemon to
> 200 Mb (live -110Mb) , and mallinfo show decreasing of total size of memory
> allocated by mmap. The daemon keeps growing, but it allocates the memory to
> arena (using brk), and calls of GC.compact leads to decrease of heap_bytes
> to 200M, but arena size (1.5 Gb) does not decrease, just doing less
> uordblks and more fordblks (fileds I.e., after first calls of GC.compact,
> the daemon starts using brk much more than mmap, and cant memory given back
> to the system.

Alternative malloc implementation may well be worth a try. I've seen the cases
when glibc was constantly growing RSS while tcmalloc was on stable level (and 10x less), 
guessing due to fragmentation.
Maybe related, see PR#5389 for ocaml compaction related issues.
Also have a look at http://ygrek.org.ua/p/code/mlvalues.py for the way to peep
into ocaml heap at runtime.

-- 
 ygrek
 http://ygrek.org.ua

Re: [Caml-list] Memory fragmentation

[rixed]

There is a commit dating from a few days ago that might
be related to your problem (commit msg was :
PR#5389: compaction sometimes leaves a very large heap)

Re: [Caml-list] Memory fragmentation

[SerP]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

I replaced call of 'malloc' in 'caml_alloc_for_heap' function to mmap, and
'free' in 'caml_free_for_heap' to unmap. Unused memory is freed after
Gc.compaction.

On Sat, Apr 21, 2012 at 11:26 AM, <rixed@happyleptic.org> wrote:

> There is a commit dating from a few days ago that might
> be related to your problem (commit msg was :
> PR#5389: compaction sometimes leaves a very large heap)
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/wws/info/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

[-- Attachment #2: Type: text/html, Size: 1251 bytes --]