From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id q08NQK6A004835 for ; Mon, 9 Jan 2012 00:26:20 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AnIBACglCk/RVdi2kGdsb2JhbABCrDwIIgEBAQEJCQ0HFAQhgXIBAQEDARICJjEOBQsLGC40AQUBHAYTGgEHh1iYTwqcIosuYwSIOYxQhVGIMT2EGA X-IronPort-AV: E=Sophos;i="4.71,476,1320620400"; d="scan'208";a="126027018" Received: from mail-qy0-f182.google.com ([209.85.216.182]) by mail4-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 09 Jan 2012 00:26:14 +0100 Received: by qcse13 with SMTP id e13so2423014qcs.27 for ; Sun, 08 Jan 2012 15:26:13 -0800 (PST) Received: by 10.224.184.79 with SMTP id cj15mr4880692qab.78.1326065172783; Sun, 08 Jan 2012 15:26:12 -0800 (PST) Received: from [192.168.1.6] (66-189-25-86.dhcp.oxfr.ma.charter.com. [66.189.25.86]) by mx.google.com with ESMTPS id m20sm138950170qaj.14.2012.01.08.15.26.11 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 08 Jan 2012 15:26:11 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: orbitz@ezabel.com In-Reply-To: <20120108230238.GB23941@annexia.org> Date: Sun, 8 Jan 2012 18:26:09 -0500 Cc: david.baelde@ens-lyon.org, Caml List Message-Id: <39EF5A76-D42F-490E-A70C-2C0EBBB34AEB@ezabel.com> References: <96F225D0-B458-4E25-BE34-3976989984B2@ezabel.com> <20120101124454.GA12851@annexia.org> <012932EC-860F-4A40-98D1-E4B6EC123927@ezabel.com> <20120108184505.GA30498@annexia.org> <20120108230238.GB23941@annexia.org> To: "Richard W.M. Jones" X-Mailer: Apple Mail (2.1084) Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by walapai.inria.fr id q08NQK6A004835 Subject: Re: [Caml-list] Understanding usage by the runtime In this case, as far as I know, all of the memory I was creating was under Ocaml, not a C extension which is why I would have expected the memory to be given back to malloc which would then give it back to the OS. I understand that the malloc implementation might decide to retain some data, the overhead of 400x the amount of active data just startled me and I feel like something else was going on. I used your maps.pl script and what the runtime seemed to be doing is growing the anonymous mapped region and then moving it into the heap and shrinking anonymous mapped region, so in my case the heap grew to 1 Gig then down to 450 megs as teh Gc could finally free the data. I don't know what was really going on under the hood though and unfortunately not sure how to figure it out. Thankfully, at this point it's just a curiosity not a production problem. Thank you On Jan 8, 2012, at 6:02 PM, Richard W.M. Jones wrote: > On Sun, Jan 08, 2012 at 05:50:40PM -0500, orbitz@ezabel.com wrote: >> Isn't the goal of compaction to keep all of these blocks of memory >> as close as possible? I should have noted the fragmentation of my >> heap after compaction, but it seems unlikely that my < 1meg of actual >> data could be fragmented across 400megs worth of chunks. > > I might not have been clear: memory can only be given back to the OS > at the C / malloc allocator level. OCaml compaction has nothing to do > with this because C allocations (and data structures used by malloc > itself) cannot ever be moved. > > However you can get a clearer picture if you look at /proc//maps > or smaps and also if you have a debugging malloc implementation. > >>> Here's the news: the OS doesn't need you to give back >>> memory. Because of virtual memory and swap, the OS will quite happily >>> take back your memory whenever it wants without asking you. >> >> For most cases this is true, however in my case (which is not the >> usual case), my OS has no swap. We actually prefer things to fail >> than to be swapped because we are doing computations could take months >> if we get into a swapping situation. I'm no linux expert so our >> solution is to not have swap and to keep our VMs light when it comes >> to I/O. Perhaps this is a poor solution but it does change things for >> us. > > Sure, this is a perfectly valid case, we have many customers who use > RHEL like this. > > Rich. > > -- > Richard Jones > Red Hat