From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail4-relais-sop.national.inria.fr (mail4-relais-sop.national.inria.fr [192.134.164.105]) by walapai.inria.fr (8.13.6/8.13.6) with ESMTP id pB8AXBfL019319 for ; Thu, 8 Dec 2011 11:33:20 +0100 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AjABAE2R4E7ZSMDqi2dsb2JhbABDDqpXIgEBAQoLCwcSBSKBcgEBBAEyAUYFCwshJQ8BBCghE4gHArZ9BIs7BJoijDg4 X-IronPort-AV: E=Sophos;i="4.71,319,1320620400"; d="scan'208";a="122564383" Received: from fmmailgate03.web.de ([217.72.192.234]) by mail4-smtp-sop.national.inria.fr with ESMTP; 08 Dec 2011 11:33:20 +0100 Received: from moweb002.kundenserver.de (moweb002.kundenserver.de [172.19.20.108]) by fmmailgate03.web.de (Postfix) with ESMTP id 61D6B1AB40687 for ; Thu, 8 Dec 2011 11:33:19 +0100 (CET) Received: from frosties.localnet ([95.208.118.96]) by smtp.web.de (mrweb002) with ESMTPA (Nemesis) id 0MRD4x-1RCRQP23EF-00UaSi; Thu, 08 Dec 2011 11:33:19 +0100 Received: from mrvn by frosties.localnet with local (Exim 4.77) (envelope-from ) id 1RYbHw-0002nY-Td; Thu, 08 Dec 2011 11:33:17 +0100 From: Goswin von Brederlow To: William Le Ferrand Cc: caml users References: Date: Thu, 08 Dec 2011 11:33:16 +0100 In-Reply-To: (William Le Ferrand's message of "Wed, 7 Dec 2011 20:35:29 -0800") Message-ID: <871usf4akz.fsf@frosties.localnet> User-Agent: Gnus/5.110009 (No Gnus v0.9) XEmacs/21.4.22 (linux, no MULE) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Provags-ID: V02:K0:Zkj2FtlwKlbnE/3l6Q1/fStNp8j0nKzxs85xdrSWdwG ziQc2VXeoA1jdxLLV1aPYRZ9yzYEtDP8aPudnttnubZhcEzKPN 1f+dFOoErW9V4vk1cEwqXvkejDQmcqviZXClzdSINGuP4sGYgA wJmlVIntIiDJWcZ+iI/XuVYRX+81r7I54+hYMVBCjHS//dFH4e k2irXS62L5YM+Nias/s3Q== Subject: Re: [Caml-list] Storing ocaml values outside ocaml's heap William Le Ferrand writes: > Dear list,  > > We are building a cache in ocaml and we're wondering if it would make sense to > store ocaml values outside the reach of the gc. (gc on a 20GB cache hangs the > process for a second or so). > > To run some experiments, we wrote a small library (https://github.com/besport/ > ocaml-everlasting) that exposes two functions, get and set.  > > When inserting a value, we copy recursively the blocs outside of the reach of > the gc (and put the resulting value in some C array). When getting the value, > we simply pass the pointer to the copied value to the ocaml code (the structure > is still coherent and the value is directly usable). We also wrote an "update" > function that compare a new value with the existing value in cache, to avoid > unnecessary memory allocation/deallocation. > > It does not seems very stable though, but I don't know if it is a bug in the > update function or simply because this approach is not reasonable. Do you have > any thoughts? Is there any clever way to build a large cache in an ocaml app ?  > > Thanks in advance for any tips! > > Best > > William For a generic case you will have to inspect the values, tell the GC about all the other ocaml values it points to and track any heap values that have a reference to your external value so you don't delete it before it is dead. In short you need to write a GC. For special cases you can store completly self contained values, e.g. a array of floats or a record of non pointer types. The simplest of types. For a cache you often have millions of objects of the same type. And often those contain pointers but do not share them. Then it becomes feasable to copy everything they point to into the cache as well. Or write a little C glue to store the data in an abstract or custom block that does not contain pointers to outside the block. Often you can also store the data much more compact than ocaml allows and thereby reduce the memory footprint and increase cache efficiency. MfG Goswin