Thanks for this insight... I imagined the lack of robustness of Marshaling,
but without all the details you mentioned!... actually, I DO desperately
need speed, as I am processing TB's of Wikipedia data, but precisely because
the datasets are so large, I cannot afford having to recompute / convert
them often, and so I want a robust format. Furthermore, I think the
bottleneck for me is anyway the speed of mysql and the disk, not really the
small amount of time that natively compiled Ocaml would take for the
conversion (I have anyway to do more complex computation that converting a
few lists and datatypes to ascii, unfortunately).  Moreover, a plaintext
format greatly helps debugging; it also helps that I can read the same data
with other programming languages.

Speaking of debugging, and said in passing, I cannot say enough how much I
LOVE the ability of ocamldebug of executing code backwards.  It is such a
revelation.  You simply go to the error, then back off a bit to see how you
got there.  But, this is a topic for another thread.

Many thanks,

Luca


On Sat, May 31, 2008 at 2:38 AM, Berke Durak <berke.durak@gmail.com> wrote:

> I second Luca's suggestion to use Sexplib.  At the very least, use a
> plaintext format.
> Don't use Marshal for long-term storage of values.  Avoid it if you
> can.  Been there, done that.
> Why?
>
> (1) Not type-safe.  Translation: your program *wil segfault* and you
> won't know why.
> (2) Not human-readable nor editable.
> (3) Not future-proof.  What happens if you change your type
> definition?  Your program
> will segfault.  So you'll have to migrate your data.  But how?  You'll
> have to find
> the exact revision used to generate the binary data.  Good luck with
> that.  Did you put
> a revision number in your data?  Are you sure it was up-to-date?  Then
> you'll have to hand-write a converter that uses type declarations from
> the old and the new modules.
> I hope your dependencies are not too complex.  Not fun *at all*.
>
> However, there are some situations where Marshal is appropriate :
>
> (1) Your data is not acyclic, contains closures, or needs sharing to
> be compact enough.  Sexplib doesn't handle these.
> (2) The data won't live long anyway.  As in: you're doing IPC between
> known versions of Ocaml programs.
> (3) You desperately need speed.  As in: you're processing 200GB of
> Wikipedia data.
> Then I can understand.
> --
> Berke Durak
>