From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (from majordomo@localhost) by pauillac.inria.fr (8.7.6/8.7.3) id RAA20729; Thu, 8 Apr 2004 17:58:05 +0200 (MET DST) X-Authentication-Warning: pauillac.inria.fr: majordomo set sender to owner-caml-list@pauillac.inria.fr using -f Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by pauillac.inria.fr (8.7.6/8.7.3) with ESMTP id RAA20677 for ; Thu, 8 Apr 2004 17:58:04 +0200 (MET DST) Received: from pauillac.inria.fr (pauillac.inria.fr [128.93.11.35]) by concorde.inria.fr (8.12.10/8.12.10) with ESMTP id i38Fw3YM005737; Thu, 8 Apr 2004 17:58:03 +0200 Received: from bourg.inria.fr (bourg.inria.fr [128.93.11.100]) by pauillac.inria.fr (8.7.6/8.7.3) with ESMTP id RAA20702; Thu, 8 Apr 2004 17:58:03 +0200 (MET DST) Received: from starynke by bourg.inria.fr with local (Exim 4.31) id 1BBbuG-0006Ve-RD; Thu, 08 Apr 2004 17:57:32 +0200 Date: Thu, 8 Apr 2004 17:57:32 +0200 To: Sebastien Ferre , caml-list@inria.fr Subject: Re: [Caml-list] swapping large data structures from/to files Message-ID: <20040408155732.GA24949@bourg.inria.fr> References: <407566DF.3040402@aber.ac.uk> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <407566DF.3040402@aber.ac.uk> User-Agent: Mutt/1.5.5.1+cvs20040105i From: Basile Starynkevitch X-Miltered: at concorde by Joe's j-chkmail ("http://j-chkmail.ensmp.fr")! X-Loop: caml-list@inria.fr X-Spam: no; 0.00; caml-list:01 basile:01 basile:01 2004:99 dags:01 persistency:01 ndbm:01 persistency:01 abort:01 unload:01 schema:01 cristal:01 cristal:01 ocaml:01 ocaml:01 Sender: owner-caml-list@pauillac.inria.fr Precedence: bulk X-Status: X-Keywords: X-UID: 156 On Thu, Apr 08, 2004 at 02:51:11PM +0000, Sebastien Ferre wrote: > I am interested in handling so large data structures > that they don't fit in main memory. I am curious - what are your huge DAGs? (bio-informatics applications??) > I need 2 things: > 1. Persistency of the data structure, preferably in > a file (similarly to NDBM, say). Did you look into Persil on my home page (see my sig)? It does provide persistency into small segmented files (which works reasonably for small data, since the whole file gets copied at end of process) or with MySQL4. If you need, I could add another persistent store for it (but I think that using a transactional database with Persil is much better for big persistent data). The most important issue is: do you need some kind of transaction mechanism? I could write some better file based persistent store iff you don't need [nested] transactions (with commit & abort ability)! You might also use Bigarray-s which can be mapped to files. > 2. Customized swapping strategy of elements of the data > structure, what should be more efficient than the > virtual memory. I'm not sure to fully understand your point. Persil does give the ability to unload & relead persistent values on (explicit) demand. Do you agree to explicitly say in your application (by appropriate calls) I won't need any more this and this values? Or do you want the system to guess them by yourself. (For completness, you can give hints to the VM system with the madvise system call, but it won't work with Ocaml - because values may be moved by the GC). > > Typically, my data structure is a DAG, and I wish to > keep in memory only a limited amount of nodes at a time. Is schema evolution a concern for you? Ie if you change the types implementing your DAG, how do you deal with the huge persistent data in that case? (Persil does not handle this issue, since it uses the Marshal module) > Hence the necessaty for swapping. It is also important > to have as much as possible in memory, and not merely > accessing the file, for efficiency reasons. > Has anything be done in this direction ? > The library Dbm is fine to me for the persistency, > but it does not work on every platform :-(. > ( Would Dbm be difficult to rewrite in OCaml ?) I think that there are quite portable versions of Dbm (or BSD DB). > Sébastien Ferré (You can answer me in French if you wish; if you CC the list, let's continue in english) -- Basile STARYNKEVITCH -- basile dot starynkevitch at inria dot fr Project cristal.inria.fr - phone +33 1 3963 5197 - mobile 6 8501 2359 http://cristal.inria.fr/~starynke --- all opinions are only mine ------------------- To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners