Re: [Caml-list] appending data to a mmap-ed file

[Jesper Louis Andersen <jesper.louis.andersen@gmail.com>]

On Thu, Dec 16, 2010 at 12:31, Joel Reymont <joelr1@gmail.com> wrote:
> I'm constantly appending to a file of stock quotes (ints, longs, doubles, etc.). I have this file mapped into memory with mmap.

Ok, this helps a bit on what you are trying to do (you asked almost
the same question on the Erlang mailing list, but the details of
getting a foothold for the same thing in Erlang is subtly different)

My approach would be simple by noting you you two kinds of data and
some peculiar behaviour
  * "Newly generated data"
  * "Old data for archeology"
  * Data are almost never deleted

So:
  * If data is less than a threshold in size (preferably less than a
couple of PAGE_SIZE page boundaries, keep data in memory and serve it
from there. Simply have an Ocaml array of bytes or something such to
store data into (my Ocaml representation specific knowledge is not up
to par at the moment, but arrange it such that the byte-array has
C-representation underneath. I know that Ocaml strings have this).
This is the newly generated data.
 * Once in a while, you write(2) this string to the file on the disk,
then reopen the mmap() (which is now READ-ONLY as an effect. There
might be sharing tricks to play here should you do multi-process).
 * Lookup is handled by checking if data is archeology or data are
recent. The right lookup is then made. Everything hidden by batching
it up in a module.
 * You can play with the factor of when to write data to disk. Too
large, and you risk loosing too much data on failure. Too small and
the approach dies of syscall-overhead.

You may have additional constraints, so spill them, please.


-- 
J.