Re: [Caml-list] Memory fragmentation

[ygrek <ygrekheretix@gmail.com>]

On Fri, 20 Apr 2012 19:12:12 +0400
SerP <serp256@gmail.com> wrote:

> Hi!
> We have developped a daemon on ocaml using the lwt lib(libev). It processes
> about 800 requests per second, but it increases to 2 Gb in memory for a
> hour of work. We have studied the problem for a long time, using mtrace,
> mallinfo and other tools, and we tried to change GC params. We found out
> that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will
> make the daemon grow slower. mallinfо shows that the memory (1,5G) is
> allocated using mmap (blkhd field in mallinfo struct) and arena size -
> about 20M In this case, first calls of GC.compact compress the daemon to
> 200 Mb (live -110Mb) , and mallinfo show decreasing of total size of memory
> allocated by mmap. The daemon keeps growing, but it allocates the memory to
> arena (using brk), and calls of GC.compact leads to decrease of heap_bytes
> to 200M, but arena size (1.5 Gb) does not decrease, just doing less
> uordblks and more fordblks (fileds I.e., after first calls of GC.compact,
> the daemon starts using brk much more than mmap, and cant memory given back
> to the system.

Alternative malloc implementation may well be worth a try. I've seen the cases
when glibc was constantly growing RSS while tcmalloc was on stable level (and 10x less), 
guessing due to fragmentation.
Maybe related, see PR#5389 for ocaml compaction related issues.
Also have a look at http://ygrek.org.ua/p/code/mlvalues.py for the way to peep
into ocaml heap at runtime.

-- 
 ygrek
 http://ygrek.org.ua