From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id q3KLZ0rs007797 for ; Fri, 20 Apr 2012 23:35:00 +0200 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvcDAOvVkU9KfVK2kGdsb2JhbABEhWiqQwSBBQgiAQEBAQkJDQcUBCOCCQEBAQMBEgIPDwENARseAwELBgULDwIYDgICIQIRAQUBHBkih14BAwYFBAecMwqLUlCCc4UkChknDVeIdgEFC4EkgmeFX4ESCoJ3gg2BGASIXYUyh2kBgRGKK4MhPYQN X-IronPort-AV: E=Sophos;i="4.75,455,1330902000"; d="scan'208";a="140990527" Received: from mail-we0-f182.google.com ([74.125.82.182]) by mail4-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 20 Apr 2012 23:34:55 +0200 Received: by wern13 with SMTP id n13so10608484wer.27 for ; Fri, 20 Apr 2012 14:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding; bh=nQS0swK30LdtY34lKnRQGrJp5cyd/zG9Z+5Zpbn7PlM=; b=zRN2S+98uofKY6U8dxrpC2n3vW663NyU2trGX8Q+7fzpugtRnzsdmHY7hA/IyKlA95 3iMlROE/3ixlbe47q9vkhG1IQlf0ei2BhJ49SwZXDNtwA6XhQDM1qMZnzE+CQ5Yx0AWx 7KAfyKNrf7Q+L6vG6lk3i3LAYFIYZc/g9wt4snLrCDz2K9B2wPJuw1y6gZ6aWfSzDAR9 v6uca7csBbkRlmNfHIj+37KCalEpNWNNDWZyJ45KFY1OmOal7DKUaYv4TQl+aqDtGPY5 nkkBbWzzBrqTMU7ePDuGJVKAKDfhUdQ6069i3nF0+L7K7BD6w2RWy8GEvE8SMGNKJLPX gvFw== Received: by 10.216.132.6 with SMTP id n6mr4962492wei.26.1334957695105; Fri, 20 Apr 2012 14:34:55 -0700 (PDT) Received: from cherry.local.tld ([213.169.92.119]) by mx.google.com with ESMTPS id gg2sm669941wib.7.2012.04.20.14.34.53 (version=SSLv3 cipher=OTHER); Fri, 20 Apr 2012 14:34:53 -0700 (PDT) Date: Sat, 21 Apr 2012 00:32:19 +0300 From: ygrek To: caml-list@inria.fr Message-Id: <20120421003219.ce3d8b2c.ygrekheretix@gmail.com> In-Reply-To: References: X-Mailer: Sylpheed 3.0.2 (GTK+ 2.20.1; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: Re: [Caml-list] Memory fragmentation On Fri, 20 Apr 2012 19:12:12 +0400 SerP wrote: > Hi! > We have developped a daemon on ocaml using the lwt lib(libev). It processes > about 800 requests per second, but it increases to 2 Gb in memory for a > hour of work. We have studied the problem for a long time, using mtrace, > mallinfo and other tools, and we tried to change GC params. We found out > that setting up GC.minor_heap_size=10Mb а major_heap_increment=2Мb will > make the daemon grow slower. mallinfо shows that the memory (1,5G) is > allocated using mmap (blkhd field in mallinfo struct) and arena size - > about 20M In this case, first calls of GC.compact compress the daemon to > 200 Mb (live -110Mb) , and mallinfo show decreasing of total size of memory > allocated by mmap. The daemon keeps growing, but it allocates the memory to > arena (using brk), and calls of GC.compact leads to decrease of heap_bytes > to 200M, but arena size (1.5 Gb) does not decrease, just doing less > uordblks and more fordblks (fileds I.e., after first calls of GC.compact, > the daemon starts using brk much more than mmap, and cant memory given back > to the system. Alternative malloc implementation may well be worth a try. I've seen the cases when glibc was constantly growing RSS while tcmalloc was on stable level (and 10x less), guessing due to fragmentation. Maybe related, see PR#5389 for ocaml compaction related issues. Also have a look at http://ygrek.org.ua/p/code/mlvalues.py for the way to peep into ocaml heap at runtime. -- ygrek http://ygrek.org.ua