From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by yquem.inria.fr (Postfix) with ESMTP id CF067BC57 for ; Wed, 1 Sep 2010 18:44:47 +0200 (CEST) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: ArgBACYgfkzRVaA0iGdsb2JhbACgWAgVAQEBExMRAx+kPIkughWGMy6IVAEBAwWFNASKFA X-IronPort-AV: E=Sophos;i="4.56,305,1280700000"; d="scan'208";a="66535906" Received: from mail-pw0-f52.google.com ([209.85.160.52]) by mail1-smtp-roc.national.inria.fr with ESMTP; 01 Sep 2010 18:44:47 +0200 Received: by pwj10 with SMTP id 10so3085675pwj.39 for ; Wed, 01 Sep 2010 09:44:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=jPf94HEFBB2gAYxp0NkSo923lUcswBmIm0gyhvFBwPw=; b=JzPzMksej4xMXm263vFEjX3dGsCGIDGDLoGAJONxGW9uYJcJ0i5KP53P4TWUtLEyQm /ERCRIxphMnfF6p5WurTIzEPIJ5lZYRao4GeXIgSAmzDJiHM/nzIjghzaKKF1BmjbRas zLQmb83Y55wwtVo/1cTHl5am6ccGE6ZvkZEhA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=gOZW10YlBtHNoMzsQRRDVNL5gjdiAmFJzzd8H/K2Bu7eoiJXv/ncSX37UBEX88A2kK OiUmYtfCpk7K6/8LmQ/z1EGmBkBW+9M5xi881nXD29qMOXd9NyJvpYghix49Z6nqjPlQ UjVUBtNtBb3+Tsj5nA0/J0x3ipf/l6FbG2loQ= MIME-Version: 1.0 Received: by 10.114.121.16 with SMTP id t16mr9108561wac.169.1283359484916; Wed, 01 Sep 2010 09:44:44 -0700 (PDT) Received: by 10.220.106.8 with HTTP; Wed, 1 Sep 2010 09:44:14 -0700 (PDT) In-Reply-To: References: <1281855821.2115.4.camel@glinka> <1281863810.2115.11.camel@glinka> Date: Wed, 1 Sep 2010 18:44:14 +0200 Message-ID: Subject: Re: [Caml-list] More re GC hanging From: Adrien To: Damien Doligez Cc: caml-list caml-list Content-Type: text/plain; charset=ISO-8859-1 X-Spam: no; 0.00; damien:01 damien:01 iirc:01 memoized:01 basile:01 bindings:01 bindings:01 'print:01 endline:01 pointer:01 alignment:01 stubs:01 doligez:01 doligez:01 garbage:01 On 01/09/2010, Damien Doligez wrote: > > On 2010-08-15, at 12:45, Adrien wrote: > >> First, remove all non-tail-rec functions: no more List.map, @ or >> List.concat. All lists were pretty short (definitely less than 1000 >> elements) but maybe the amount of calls generated garbage or something >> like that: I couldn't get much infos about the problem so I tried what >> I could think of and being tail-rec couldn't be a bad thing anyway. >> The idea was to create less values since I first suspected a memory >> fragmentation issue (iirc I had thousands of fragments), so I also >> memoized some functions. > > That has nothing to do with the GC getting huge counts. I know but I first had crashes which didn't show the huge counts and did what I had planned to do for some time. Also, I was actually generating lots of garbage (well, maybe not 10^20 ;-)). > Also, if you > have fragmentation problems, you can try changing the allocation > policy with this statement: > > Gc.set {(Gc.get ()) with Gc.allocation_policy = 1};; > > I'm still waiting for feedback on that alternate allocation policy :-) I had tried that, it didn't change anything. >> Then, as Basile mentionned, call something like Gc.compact () >> regularly. The overhead is actually pretty small, especially if ran >> regularly. > > That's good for tracking down problems, but I wouldn't recommend it > for production code. > >> Finally, C bindings. I created a few while not having access to the >> internet and they are quite dirty. I highly doubt they play perfectly >> well with the garbage collector: they seem ok but probably aren't >> perfect. That's definitely something to check, even if the bindings >> were written by someone else because working nicely with the GC can be >> quite hard. >> >> Now, the problem seems to be gone but I can't say for sure. One really >> annoying thing was that adding a line like 'print_endline "pouet";' >> would make the out-of-memory problem go away! Same when getting stats >> from the GC. > > > That almost certainly indicates a problem with your C bindings: some > pointer gets garbled and the GC may or may not stumble upon it. That's also what I think: calling Gc.compact () doesn't solve the problem, it only changes the planet alignment and the phase of the moon. Sorry for the late reaction, I was pretty short on time during the past ten days but it's going to be better now. :-) I took a quick look at the C stubs and noticed a few variables of type 'value' where not introduced with CAMLlocalX(), in particular the creation of a list. I don't know if that's enough to fix the problem since it wasn't happening anymore on my computer and I'm now waiting for someone to be able to test. -- Adrien Nader