From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id q075hTLA030218 for ; Sat, 7 Jan 2012 06:43:30 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhwBAMvaB0/RVdi2kGdsb2JhbABErDcIIgEBAQEJCQ0HFAQhgXIBAQEDARICJj8FCwsYLjQBBQEcBhMih1iZKgqcTIsuYwSIOYxPhVGILz2EGA X-IronPort-AV: E=Sophos;i="4.71,472,1320620400"; d="scan'208";a="138215897" Received: from mail-qy0-f182.google.com ([209.85.216.182]) by mail1-smtp-roc.national.inria.fr with ESMTP/TLS/RC4-SHA; 07 Jan 2012 06:43:27 +0100 Received: by qcse13 with SMTP id e13so1722260qcs.27 for ; Fri, 06 Jan 2012 21:43:26 -0800 (PST) Received: by 10.224.220.14 with SMTP id hw14mr11040105qab.42.1325915006471; Fri, 06 Jan 2012 21:43:26 -0800 (PST) Received: from [192.168.1.6] (66-189-25-86.dhcp.oxfr.ma.charter.com. [66.189.25.86]) by mx.google.com with ESMTPS id co15sm30580857qab.1.2012.01.06.21.43.24 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 06 Jan 2012 21:43:24 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: orbitz@ezabel.com In-Reply-To: <20120101124454.GA12851@annexia.org> Date: Sat, 7 Jan 2012 00:43:22 -0500 Cc: david.baelde@ens-lyon.org, Caml List Message-Id: <012932EC-860F-4A40-98D1-E4B6EC123927@ezabel.com> References: <96F225D0-B458-4E25-BE34-3976989984B2@ezabel.com> <20120101124454.GA12851@annexia.org> To: "Richard W.M. Jones" X-Mailer: Apple Mail (2.1084) Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by walapai.inria.fr id q075hTLA030218 Subject: Re: [Caml-list] Understanding usage by the runtime Hello everyone! I would like to reassure you that all is right in the world. After a large number of tests I finally tracked the problem down to an entry in a Hashtbl not being deleted. It was a one line fix! One question does remain though: In my tests I would do some work that would cause ~1GB of RAM to be under control of the Gc. Then I would do something that, at the time I didn't understand, would case the Gc to compact all of its memory and go back down to less than 1 meg, but the RES value in top would only drop to about 400 megs. Is this expected behavior? I know the malloc implementation might hold on to some data for itself but 400x the amount of memory the Ocaml RTS actually needs seems a bit excessive. I know there is a bug report floating around from Martin about large Arrays not being properly freed, in this case my issue was with a Hashtbl. I do not know if Hashtbl is implemented with an Array underneath, but could that be the cause of my overhead if so? Thank you On Jan 1, 2012, at 7:44 AM, Richard W.M. Jones wrote: > On Sat, Dec 31, 2011 at 10:33:19AM -0500, orbitz@ezabel.com wrote: >> Being on the C side is not even something I had considered. In this >> case, I think the only piece of code not part of the Ocaml RTS that is >> talking to C is Lwt. It is possible that there is a memory leak in >> there somewhere. The upside, though, is there seems to be some >> residue of it in the Ocaml side. My heap numbers given earlier are >> ~65megs which is significantly larger than it should be, so I might be >> able to track it down from the Ocaml side. > > A couple of other ideas: > > Is compaction disabled? lablgtk disables it unconditionally by > setting the global Gc max_overhead (see also the Gc documentation): > > src/gtkMain.ml: > let () = Gc.set {(Gc.get()) with Gc.max_overhead = 1000000} > > If something in your program or Lwt does the same, you may get > fragmentation of the C malloc heap or perhaps the OCaml heap. I've > experienced fragmentation in very long-running C programs and it's > insidious because it's very hard to understand what's really going on, > and impossible IME to remedy it. > > Second suggestion is to look at /proc//maps and/or smaps. > That'll tell you without doubt where the 2GB of memory is being used. > Most likely in the heap from the way you describe it, but it is worth > checking that top isn't reporting something innocuous such as a big > file-backed mmap in one of your C libraries. > > Attached is a script that you can adapt to help you interpret > /proc//maps. > > Rich. > > -- > Richard Jones > Red Hat >