From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (from majordomo@localhost) by pauillac.inria.fr (8.7.6/8.7.3) id GAA13840; Mon, 22 Mar 2004 06:05:28 +0100 (MET) X-Authentication-Warning: pauillac.inria.fr: majordomo set sender to owner-caml-list@pauillac.inria.fr using -f Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by pauillac.inria.fr (8.7.6/8.7.3) with ESMTP id GAA14186 for ; Mon, 22 Mar 2004 06:05:26 +0100 (MET) Received: from gum.itee.uq.edu.au (gum.itee.uq.edu.au [130.102.66.1]) by concorde.inria.fr (8.12.10/8.12.10) with ESMTP id i2M55NHd008019 for ; Mon, 22 Mar 2004 06:05:25 +0100 Received: from luma.itee.uq.edu.au (luma.itee.uq.edu.au [130.102.66.14]) by gum.itee.uq.edu.au (8.12.10/8.12.10) with ESMTP id i2M558Us023075 for ; Mon, 22 Mar 2004 15:05:08 +1000 (EST) Received: from itee.uq.edu.au (g613-8948.itee.uq.edu.au [130.102.66.112]) by luma.itee.uq.edu.au (8.12.10/8.12.9) with ESMTP id i2M558qd014577 for ; Mon, 22 Mar 2004 15:05:08 +1000 (EST) Message-ID: <405E7404.3050306@itee.uq.edu.au> Date: Mon, 22 Mar 2004 15:05:08 +1000 From: Richard Cole User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.5) Gecko/20031016 X-Accept-Language: en-us, en MIME-Version: 1.0 To: caml-list@inria.fr Subject: [Caml-list] Memory Mapped Files and OCaml Content-Type: multipart/alternative; boundary="------------070208090707090801030702" X-Spam-Checked: This message probably not SPAM X-Spam-Score: 1.4 X-Spam-Level: * (1.4) X-Spam-Tests: HTML_10_20,HTML_MESSAGE,USER_AGENT_MOZILLA_UA,X_ACCEPT_LANG X-Spam-Report: 1.40 points, 8 required; * 0.0 -- User-Agent header indicates a non-spam MUA (Mozilla) * -0.1 -- Has a X-Accept-Language header * 1.4 -- BODY: Message is 10% to 20% HTML * 0.1 -- BODY: HTML included in message X-Scanned-By: MIMEDefang 2.35 X-Miltered: at concorde by Joe's j-chkmail ("http://j-chkmail.ensmp.fr")! X-Loop: caml-list@inria.fr X-Spam: no; 0.00; cole:99 pointers:01 thrashing:01 conjuction:01 referential:01 backends:01 pointers:01 thrashing:01 conjuction:01 referential:01 backends:01 inconsistent:01 inconsistent:01 ocaml:01 ocaml:01 Sender: owner-caml-list@pauillac.inria.fr Precedence: bulk X-Status: X-Keywords: X-UID: 295 This is a multi-part message in MIME format. --------------070208090707090801030702 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Hi, I wonder if anyone can give me some pointers. I'm interested in having all memory used by my ocaml program memory mapped so that calculations can be preserved from one run of an ocaml program to the next. I'm thinking of something like: let empty_list : (string, string) list = [] in *let user_list = store.get_value "user_list" empty_list in* let user : string = Cgi.get_value "user" in let password : string = Cgi.get_value "password" in if exists user user_list then if auth user password user_list then print_account_details user else print_account_details user else begin *store.put_value "user_list" (add_user user password user_list);* print_account_details user end With the idea being that all values are stored in the memory mapped files so put_value and get_value are very fast. Serialization is ok for small data structures, but for 50M data structures, for which only a small part of the data structure is accesed, it is a pain. The idea is that put_value and get_value cause structures to survive to the uppermost scope level and so these are not garbage collected before program termination since they are still referenced. Everything else gets garbage collected and so doesn't clog up the persistent store. Of course program termination can take a long time if there are many dirty pages that need to be synchronised to disk. There may be some way to tell unix to sync dirty pages while the program is running but without thrashing (i.e. using all system resources). Such a persistent store does suffer from a lack of safety. i.e. killing the process or the machine going down could leave the store in an inconsistent state. If safety is required there must be algorithms around to provide it in conjuction with a memory mapped file, perhaps via checkpointing. Does referential transparency help us here? One final question: Are most people using database backends for persistence? Is it the case that most data structures that one would want to create in Ocaml programs map fairly easily into B-tree structures, i.e. are maps or multimaps from a keyed domain into some structured domain. best regards, Richard. --------------070208090707090801030702 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit Hi,

I wonder if anyone can give me some pointers. I'm interested in having all memory used by my ocaml program memory mapped so that calculations can be preserved from one run of an ocaml program to the next. I'm thinking of something like:

let empty_list : (string, string) list = [] in
let user_list = store.get_value "user_list" empty_list in
let user : string = Cgi.get_value "user" in
let password : string = Cgi.get_value "password" in
  if exists user user_list then
    if auth user password user_list then
      print_account_details user       
    else
      print_account_details user
  else
    begin
      store.put_value "user_list" (add_user user password user_list);
      print_account_details user
    end 

With the idea being that all values are stored in the memory mapped files so put_value and get_value are very fast. Serialization is ok for small data structures, but for 50M data structures, for which only a small part of the data structure is accesed, it is a pain.

The idea is that put_value and get_value cause structures to survive to the uppermost scope level and so these are not garbage collected before program termination since they are still referenced. Everything else gets garbage collected and so doesn't clog up the persistent store.

Of course program termination can take a long time if there are many dirty pages that need to be synchronised to disk. There may be some way to tell unix to sync dirty pages while the program is running but without thrashing (i.e. using all system resources).

Such a persistent store does suffer from a lack of safety. i.e. killing the process or the machine going down could leave the store in an inconsistent state. If safety is required there must be algorithms around to provide it in conjuction with a memory mapped file, perhaps via checkpointing. Does referential transparency help us here?

One final question: Are most people using database backends for persistence? Is it the case that most data structures that one would want to create in Ocaml programs map fairly easily into B-tree structures, i.e. are maps or multimaps from a keyed domain into some structured domain.

best regards,

Richard.

--------------070208090707090801030702-- ------------------- To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners