From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=AWL autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by yquem.inria.fr (Postfix) with ESMTP id 37597BC6B for ; Fri, 17 Aug 2007 17:00:31 +0200 (CEST) Received: from furbychan.cocan.org (furbychan.cocan.org [80.68.91.176]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id l7HF0UvG005176 (version=TLSv1/SSLv3 cipher=EDH-RSA-DES-CBC3-SHA bits=168 verify=NO) for ; Fri, 17 Aug 2007 17:00:30 +0200 Received: from rich by furbychan.cocan.org with local (Exim 3.35 #1 (Debian)) id 1IM3JN-00048W-00 for ; Fri, 17 Aug 2007 16:00:29 +0100 Date: Fri, 17 Aug 2007 16:00:29 +0100 To: caml-list@inria.fr Subject: Re: [Caml-list] Stopping and continuing GC Message-ID: <20070817150029.GA5052@furbychan.cocan.org> References: <20070817.100453.94562786.garrigue@math.nagoya-u.ac.jp> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.9i From: Richard Jones X-Miltered: at concorde with ID 46C5B80E.003 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; markus:01 mottl:01 compaction:01 compaction:01 compactions:01 byterun:01 compactor:01 allocating:01 ocaml:01 malloc:01 allocator:01 ocaml:01 allocations:01 malloc:01 alloc:01 On Fri, Aug 17, 2007 at 10:05:41AM -0400, Markus Mottl wrote: > This, btw., raises an interesting question: are there any ways (and > hopefully even intentions) to make compaction work in slices like the > major GC does? We write extremely latency sensitive applications and > often even have to manually control when the GC performs collections. > > Unfortunately, compaction cannot be controlled in the same way as the > major GC can be (e.g. calling the major GC with a specific slice > size). That's why we have to turn off automatic compactions, too, and > run these latency sensitive applications on machines with very large > amounts of memory. I looked at the code in byterun/compact.c (I've been looking at the internals of the GC quite a bit this week) and at first glance it seems like it overwrites headers in the heap temporarily while compacting. So any incremental compactor looks like it would involve quite a rewrite. But I wonder in general terms about the problem you are having, which is surely a fragmentation issue (exactly the same as you'd see in a C program). Can this be cured by examining what sizes of structures you are allocating? Perhaps you can pad out or trim common structures so that they are the same size, thus avoiding fragmentation? A second possibility is that it's an interaction between the OCaml heap blocks and the underlying C malloc allocator. It might be that OCaml is requesting allocations which always require more memory from the operating system (ie. they are too large or awkwardly sized to satisfy from free areas within the current C heap). It should be relatively easy to diagnose either situation: For the first you'd need a program to walk the Caml heap looking at the sizes of blocks, which you'd run in a process that has been running for some time. For the second, instrument calls to malloc (in caml_alloc_for_heap). That's just from a cursory look at the code anyway -- there is probably much that I'm missing, but instrumenting those would be a start. Rich. -- Richard Jones Red Hat