Persistent storage and stability of Marshal?

Persistent storage and stability of Marshal?

[David MENTRE]

Hello,

I need to implement some kind of persistent storage in my application
(demexp web proxy part and demexp server part), so I'm looking at the
different possible alternatives.

Roughly, I've found:

 - Dbm standard module of OCaml: file storage, no external dependency,
   safe?

 - Berkeley DB 4.x interface[1]: file storage, seems safe;

 - SQL database[2]: real database, complicated to setup, safe.

Right now, I don't have big needs: few tables of moderate size (below
100,000 records) so I'm in favor of simple solutions: Dbm or Berkeley DB
(no need to setup an external SQL server). I need to lookup values
through known keys. The key and stored data are simple records of
strings and integers. In the future, I might need to lookup rows
according to certain field value.

Looking at code using Berkeley DB, I noticed that stored key and data is
made using the Marshal module. However, Marshal doc says that
marshalling is "compatible across all machines for a given version of
Objective Caml", so there might be issues when changing OCaml version. 

So my questions:

 1. Is the Marshal module that much unstable? I have the feeling that
    marhsaled values are compatible between OCaml releases. True or
    false?

 2. Any advice on implementing persistent storage? I know about Persil
    library[3] but I don't see much advantage of using it (does it
    implement its own marshaling, more stable than Marshal?).

Best wishes,
d.

Footnotes: 
[1]  http://www.eecs.harvard.edu/~stein/ocamlbdb-4.3.21.tar.gz

[2]  OCamlDBI: http://caml.inria.fr/cgi-bin/hump.fr.cgi?contrib=381

[3]  http://cristal.inria.fr/~starynke/persil/

-- 
pub  1024D/A3AD7A2A 2004-10-03 David MENTRE <dmentre@linux-france.org>
 5996 CC46 4612 9CA4 3562  D7AC 6C67 9E96 A3AD 7A2A

Re: [Caml-list] Persistent storage and stability of Marshal?

[Xavier Leroy]

>  1. Is the Marshal module that much unstable? I have the feeling that
>     marhsaled values are compatible between OCaml releases. True or
>     false?

True in practice: the last incompatible change was done in 1996...
Still, the Marshal module is not intended for long-term storage of
data that you cannot easily reconstruct from other sources.  In
particular, the data format is compressed enough that it is not
possible to salvage data using, say, a text editor.  At the very
least, it is prudent to build into your software functions to dump and
restore marshaled data to/from a simpler, textual format.  (That's
what I did for SpamOracle's database, which is a marshaled hash table.)

>  2. Any advice on implementing persistent storage? I know about Persil
>     library[3] but I don't see much advantage of using it (does it
>     implement its own marshaling, more stable than Marshal?).

There are two independent questions:

1- How to encode your data into character strings?
2- How to store these strings in persistent storage?

For 2-, you have many options, ranging from flat files (the
traditional Unix way) to full-fledged databases.

For 1-, in addition to the Marshal module, you can also use textual
formats: key-value pairs, XML, Lisp S-expressions, etc.  Other
options are XDR or ASN1 encodings.

- Xavier Leroy

Re: [Caml-list] Persistent storage and stability of Marshal?

[Florian Weimer]

* David MENTRE:

> Right now, I don't have big needs: few tables of moderate size (below
> 100,000 records) so I'm in favor of simple solutions: Dbm or Berkeley DB
> (no need to setup an external SQL server). I need to lookup values
> through known keys. The key and stored data are simple records of
> strings and integers. In the future, I might need to lookup rows
> according to certain field value.

Have a look at SQLite.  I like it a lot.  During the past couple of
months, even a few Ocaml bindings have sprung into existence.

Re: [Caml-list] Persistent storage and stability of Marshal?

[David MENTRE]

Hello,

Florian Weimer <fw@deneb.enyo.de> writes:

> Have a look at SQLite.  I like it a lot.  During the past couple of
> months, even a few Ocaml bindings have sprung into existence.

Yes, I looked at it through Dbi[1]. There is however an issue: it seems
that everything is stored as a string in an SQLite base. Returned data
after Dbi query are typed as string.

It might be an issue with Dbi itself (I haven't looked to the SQLite
binding themselves) but after looking at SQLite documentation, it does
not seem to be the case[2].

I could work around this small issue by easily converting strings to
integer, float, etc. but it seems a bit awkward to me. Beside that,
you are right the SQLite is quite interesting: full SQL database but
without the need to setup an SQL server.

Best wishes,
david

Footnotes: 
[1]  http://download.savannah.nongnu.org/releases/modcaml/

[2]  http://www.sqlite.org/datatypes.html
     ""
      SQLite is "typeless". This means that you can store any kind of
      data you want in any column of any table, regardless of the
      declared datatype of that column.
     ""
-- 
pub  1024D/A3AD7A2A 2004-10-03 David MENTRE <dmentre@linux-france.org>
 5996 CC46 4612 9CA4 3562  D7AC 6C67 9E96 A3AD 7A2A

Re: [Caml-list] Persistent storage and stability of Marshal?

[Florian Weimer]

* David MENTRE:

> Hello,
>
> Florian Weimer <fw@deneb.enyo.de> writes:
>
>> Have a look at SQLite.  I like it a lot.  During the past couple of
>> months, even a few Ocaml bindings have sprung into existence.
>
> Yes, I looked at it through Dbi[1]. There is however an issue: it seems
> that everything is stored as a string in an SQLite base. Returned data
> after Dbi query are typed as string.
>
> It might be an issue with Dbi itself (I haven't looked to the SQLite
> binding themselves) but after looking at SQLite documentation, it does
> not seem to be the case[2].

It's got manifestant types.  There are strings, integers, reals and
BLOBs (and NULL, of course).  You need a binding which stores integers
as integers, and not as strings, though.  SQLite mostly ignores column
types and uses on;y the type of the value you insert into the
database.

Re: [Caml-list] Persistent storage and stability of Marshal?

[N. Owen Gunden]

On Tue, Dec 27, 2005 at 11:17:13AM +0100, Xavier Leroy wrote:
> For 1-, in addition to the Marshal module, you can also use textual
> formats: key-value pairs, XML, Lisp S-expressions, etc.  Other
> options are XDR or ASN1 encodings.

You could consider using the fantastic S-expression library, developed
at Jane Street Capital, for this step.

Here's the thread where sexplib was introduced:

http://tinyurl.com/9m4s4

 - O