From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 1F9C07F722 for ; Mon, 14 Apr 2014 22:39:13 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of rich@annexia.org) identity=pra; client-ip=80.68.91.176; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="rich@annexia.org"; x-sender="rich@annexia.org"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of rich@annexia.org designates 80.68.91.176 as permitted sender) identity=mailfrom; client-ip=80.68.91.176; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="rich@annexia.org"; x-sender="rich@annexia.org"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@furbychan.cocan.org) identity=helo; client-ip=80.68.91.176; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="rich@annexia.org"; x-sender="postmaster@furbychan.cocan.org"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AhYFACRHTFNQRFuw/2dsb2JhbABZgwaDUMBygScWdIIlAQEBBDo/EAsYCRMSDwUoIYgTAcwIF4V/iG8HgySBFASUdINskkSDMjw X-IPAS-Result: AhYFACRHTFNQRFuw/2dsb2JhbABZgwaDUMBygScWdIIlAQEBBDo/EAsYCRMSDwUoIYgTAcwIF4V/iG8HgySBFASUdINskkSDMjw X-IronPort-AV: E=Sophos;i="4.97,859,1389740400"; d="scan'208";a="57118210" Received: from furbychan.cocan.org ([80.68.91.176]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES128-SHA; 14 Apr 2014 22:38:45 +0200 Received: from rich by furbychan.cocan.org with local (Exim 4.80) (envelope-from ) id 1WZneN-0006Tc-0i; Mon, 14 Apr 2014 21:38:43 +0100 Date: Mon, 14 Apr 2014 21:38:43 +0100 From: "Richard W.M. Jones" To: Arthur Breitman Cc: caml-list@inria.fr Message-ID: <20140414203843.GA23606@annexia.org> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Subject: Re: [Caml-list] Keeping A big data optimization problem functional On Sun, Apr 13, 2014 at 11:25:18PM -0600, Arthur Breitman wrote: > So total, I'm looking at 10GB to 1TB one data which is typically stored in > a database on disk. The tree is loaded in memory using keys that represent > entries in the database. > > Another structure I'm dealing with is a key value store, whose size is on > the same order of magnitude as the size of the data above. Initially, this > key-value store is empty. It might not be suitable, but ocaml-ancient would let you keep certain OCaml structs on disk, but mmapped so you can just use them as if they are in memory. (I believe it won't actually work with Map unless you write a special comparison function, but anyway you'll need to read the README file & possibly the source closely if you go down this route.) Back in the day I used to use ocaml-ancient to do analysis on large structs of the order of 30+ GB -- large in those days ... Rich. -- Richard Jones Red Hat