[Caml-list] Memory Mapped Files and OCaml

[Richard Cole <rcole@itee.uq.edu.au>]

[-- Attachment #1: Type: text/plain, Size: 2214 bytes --]

Hi,

I wonder if anyone can give me some pointers. I'm interested in having 
all memory used by my ocaml program memory mapped so that calculations 
can be preserved from one run of an ocaml program to the next. I'm 
thinking of something like:

let empty_list : (string, string) list = [] in
*let user_list = store.get_value "user_list" empty_list in*
let user : string = Cgi.get_value "user" in
let password : string = Cgi.get_value "password" in
  if exists user user_list then
    if auth user password user_list then
      print_account_details user       
    else
      print_account_details user
  else
    begin
      *store.put_value "user_list" (add_user user password user_list);*
      print_account_details user
    end 

With the idea being that all values are stored in the memory mapped 
files so put_value and get_value are very fast. Serialization is ok for 
small data structures, but for 50M data structures, for which only a 
small part of the data structure is accesed, it is a pain.

The idea is that put_value and get_value cause structures to survive to 
the uppermost scope level and so these are not garbage collected before 
program termination since they are still referenced. Everything else 
gets garbage collected and so doesn't clog up the persistent store.

Of course program termination can take a long time if there are many 
dirty pages that need to be synchronised to disk. There may be some way 
to tell unix to sync dirty pages while the program is running but 
without thrashing (i.e. using all system resources).

Such a persistent store does suffer from a lack of safety. i.e. killing 
the process or the machine going down could leave the store in an 
inconsistent state. If safety is required there must be algorithms 
around to provide it in conjuction with a memory mapped file, perhaps 
via checkpointing. Does referential transparency help us here?

One final question: Are most people using database backends for 
persistence? Is it the case that most data structures that one would 
want to create in Ocaml programs map fairly easily into B-tree 
structures, i.e. are maps or multimaps from a keyed domain into some 
structured domain.

best regards,

Richard.


[-- Attachment #2: Type: text/html, Size: 2782 bytes --]