From: "Richard W.M. Jones" <rich@annexia.org>
To: Arthur Breitman <arthur.breitman@gmail.com>
Cc: caml-list@inria.fr
Subject: Re: [Caml-list] Keeping A big data optimization problem functional
Date: Mon, 14 Apr 2014 21:38:43 +0100 [thread overview]
Message-ID: <20140414203843.GA23606@annexia.org> (raw)
In-Reply-To: <CAAYUt0P4AzcYPbNLYsHRpCnBGa=oJT3MRfKwZv-wE4vv-JXNkw@mail.gmail.com>
On Sun, Apr 13, 2014 at 11:25:18PM -0600, Arthur Breitman wrote:
> So total, I'm looking at 10GB to 1TB one data which is typically stored in
> a database on disk. The tree is loaded in memory using keys that represent
> entries in the database.
>
> Another structure I'm dealing with is a key value store, whose size is on
> the same order of magnitude as the size of the data above. Initially, this
> key-value store is empty.
It might not be suitable, but ocaml-ancient would let you keep certain
OCaml structs on disk, but mmapped so you can just use them as if they
are in memory. (I believe it won't actually work with Map unless you
write a special comparison function, but anyway you'll need to read
the README file & possibly the source closely if you go down this
route.)
Back in the day I used to use ocaml-ancient to do analysis on large
structs of the order of 30+ GB -- large in those days ...
Rich.
--
Richard Jones
Red Hat
prev parent reply other threads:[~2014-04-14 20:39 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-14 5:25 Arthur Breitman
2014-04-14 5:43 ` Francois Berenger
2014-04-14 6:07 ` Arthur Breitman
2014-04-14 8:12 ` Goswin von Brederlow
2014-04-14 19:32 ` Arthur Breitman
2014-04-15 9:05 ` Goswin von Brederlow
2014-04-15 13:16 ` Arthur Breitman
2014-04-15 13:50 ` Thomas Gazagnaire
2014-04-16 11:02 ` Arthur Breitman
2014-04-14 20:38 ` Richard W.M. Jones [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140414203843.GA23606@annexia.org \
--to=rich@annexia.org \
--cc=arthur.breitman@gmail.com \
--cc=caml-list@inria.fr \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).