Marshalling question

Marshalling question

[Jean Krivine]

Dear ocaml users,

A simple question: is it safe to marshalize a data structure that
contains imperative elements (like arrays or hashtbl) ?
I found the documentation of the Marshal module rather obscure to me.
In particular it is not clear whether I should use the No_sharing
flag.

Thank you!
J

RE: [Caml-list] Marshalling question

[David Allsopp]

Jean Krivine wrote:
> Dear ocaml users,
> 
> A simple question: is it safe to marshalize a data structure that contains
> imperative elements (like arrays or hashtbl) ?

Simple answer: yes. Marshal works on the runtime representation of data which is imperative (immutability is enforced by the type system, not the runtime).

> I found the documentation of the Marshal module rather obscure to me.
> In particular it is not clear whether I should use the No_sharing flag.

Unless you know how the data structures you are marshalling work, you should never use this flag (i.e. if you're working with an abstract data type like Hashtbl.t then you certainly shouldn't be using this flag). The obvious reason is that if the data contain a cycle that you're not aware of then your program will not terminate. Less obvious reasons include structures (e.g. for memoizing) which rely on value sharing so that they can use physical, rather than structural equality for speed. The worst thing that can happen when this flag isn't given is that the operation might be a teensy bit slower than it strictly needs to be.



David 

Re: [Caml-list] Marshalling question

[Mathias Kende]

Le vendredi 08 octobre 2010 à 15:37 +0200, Jean Krivine a écrit :
> Dear ocaml users,
> 
> A simple question: is it safe to marshalize a data structure that
> contains imperative elements (like arrays or hashtbl) ?

It's relatively safe to do so. The only thing is that if it is
unmarshalled in the process where it was marshalled you will still and
up with a new array or hashtbl.

Exception are some complex datastructure which may require additional
care when marshalled. An example of which are the graphs of the
ocamlgraph library (even the functional one), but there is none in the
standard library.

> I found the documentation of the Marshal module rather obscure to me.
> In particular it is not clear whether I should use the No_sharing
> flag.

This flag is never needed but there is cases where it is mandatory *not*
to use it (with cyclic data).

Mathias

> 
> Thank you!
> J
> 
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] Marshalling question

[forum]

Le 8 oct. 2010 à 15:37, Jean Krivine a écrit :

> Dear ocaml users,
> 
> A simple question: is it safe to marshalize a data structure that
> contains imperative elements (like arrays or hashtbl) ?

Well, you should have no problem with arrays or hash tables,
as long as they hold elements that are themselves marshallable.

To the best of my knowledge, marshalling is either not possible or unsafe with :
  - weak array, or weak hash table ;
  - exceptions ;
  - functional values in some settings (e. g. native compilation, use of "Dynlink"),
    and hence lazy values, streams, or any functional/imperative structure
    holding a closure.
Finally, I have some slight doubts regarding objects, and first-class modules
(they should be marshallable modulo the restrictions above, but never tested
this point).


Hope this helps,

Xavier Clerc

Re: [Caml-list] Marshalling question

[Alexey Rodriguez]

On Fri, Oct 8, 2010 at 3:48 PM, Mathias Kende <mathias.kende@ens.fr> wrote:

> Exception are some complex datastructure which may require additional
> care when marshalled. An example of which are the graphs of the
> ocamlgraph library (even the functional one), but there is none in the
> standard library.

Mathias, can you elaborate on "additional care"? We are using the
functional graphs from ocamlgraph, so I am very interested in your
experiences with it.

Cheers,

Alexey

Re: [Caml-list] Marshalling question

[Julien Signoles]

[-- Attachment #1: Type: text/plain, Size: 1554 bytes --]

Hello,

2010/10/12 Alexey Rodriguez <mrchebas@gmail.com>

> On Fri, Oct 8, 2010 at 3:48 PM, Mathias Kende <mathias.kende@ens.fr>
> wrote:
>
> > Exception are some complex datastructure which may require additional
> > care when marshalled. An example of which are the graphs of the
> > ocamlgraph library (even the functional one), but there is none in the
> > standard library.
>
> Mathias, can you elaborate on "additional care"? We are using the
> functional graphs from ocamlgraph, so I am very interested in your
> experiences with it.
>

To my knowledge, issues come with the so-called "abstract" graphs from
OcamlGraph in which vertices have unique ids.
1) if there are abstract vertices in RAM while unmarshalling some others,
you have to be sure that RAM vertices do not share ids with unmarshalled
ones (except if you really want that).
2) Before creating new abstract vertices, you have to set the global id
counter Graph.Blocks.cpt_vertex in the right way (by marshalling it and by
calling Graph.Blocks.after_unserialization after unmarshalling).
I'm not aware of any other (un)marshalling issue with OcamlGraph
datastructures.

Beyond OcamlGraph, similar issues potentially exist for any datastructure
which uses unique ids as keys and/or hashconsing techniques. For instance, a
quite big job is done for solving this issue in Frama-C (http://frama-c.com)
which uses many hashconsed datastructures (included abstract ocamlgraph
vertices, but only the above-mentioned issue 2 applies in the Frama-C
context)

Hope this helps,
Julien Signoles

[-- Attachment #2: Type: text/html, Size: 2021 bytes --]

Re: [Caml-list] Marshalling question

[Mathias Kende]

Le mardi 12 octobre 2010 à 10:42 +0200, Alexey Rodriguez a écrit :
> Mathias, can you elaborate on "additional care"? We are using the
> functional graphs from ocamlgraph, so I am very interested in your
> experiences with it.

There is not much to say. To represent "abstract" graphs (those were the
equality for the nodes type is not used to check if two nodes of a graph
are the same), the library uses an internal counter. This counter must
be serialised along with the graphs and then it must be updated
correctly when graphs are unserialised to avoid creating a node with the
same identifier than o node in the unserialised graphs.

This is explained in the FAQ :
http://ocamlgraph.lri.fr/FAQ

But I bielieve that the FAQ got it wrong in case multiple graphs are
unserialised, or nodes are created before the unserialisation of other
nodes. In these situation using concrete graphs, which do not suffer
from this problem, is certainly easier (as advertised by the manual).
And I manually add identifiers to the nodes if I need many nodes with
the same label.

Mathias

> 
> Cheers,
> 
> Alexey

Re: [Caml-list] Marshalling question

[Julien Signoles]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

2010/10/12 Mathias Kende <mathias@kende.fr>

> To represent "abstract" graphs (those were the
> equality for the nodes type is not used to check if two nodes of a graph
> are the same), the library uses an internal counter. This counter must
> be serialised along with the graphs and then it must be updated
> correctly when graphs are unserialised to avoid creating a node with the
> same identifier than o node in the unserialised graphs.
>
> This is explained in the FAQ :
> http://ocamlgraph.lri.fr/FAQ
>

I didn't remember myself that the OcamlGraph FAQ contains a section about
that ;-).

But I bielieve that the FAQ got it wrong in case multiple graphs are
> unserialised, or nodes are created before the unserialisation of other
> nodes. In these situation using concrete graphs, which do not suffer
> from this problem, is certainly easier (as advertised by the manual).
> And I manually add identifiers to the nodes if I need many nodes with
> the same label.
>

Of course you're right (see also my own answer to Alexey) : I will update
the OcamlGraph FAQ accordingly.

Thanks,
Julien

[-- Attachment #2: Type: text/html, Size: 1662 bytes --]

Re: [Caml-list] Marshalling question

[Alexey Rodriguez]

Julien, Mathias,

Thanks for the detailed description of this issue.

Mathias, we use the same solution: concrete graphs and explicit
identifiers so we do not suffer from the deserialization issue. Good
to know about the pitfalls of abstract graphs though.

Cheers,

Alexey

On Tue, Oct 12, 2010 at 11:42 AM, Julien Signoles
<julien.signoles@gmail.com> wrote:
>
>
> 2010/10/12 Mathias Kende <mathias@kende.fr>
>>
>> To represent "abstract" graphs (those were the
>> equality for the nodes type is not used to check if two nodes of a graph
>> are the same), the library uses an internal counter. This counter must
>> be serialised along with the graphs and then it must be updated
>> correctly when graphs are unserialised to avoid creating a node with the
>> same identifier than o node in the unserialised graphs.
>>
>> This is explained in the FAQ :
>> http://ocamlgraph.lri.fr/FAQ
>
> I didn't remember myself that the OcamlGraph FAQ contains a section about
> that ;-).
>
>> But I bielieve that the FAQ got it wrong in case multiple graphs are
>> unserialised, or nodes are created before the unserialisation of other
>> nodes. In these situation using concrete graphs, which do not suffer
>> from this problem, is certainly easier (as advertised by the manual).
>> And I manually add identifiers to the nodes if I need many nodes with
>> the same label.
>
> Of course you're right (see also my own answer to Alexey) : I will update
> the OcamlGraph FAQ accordingly.
>
> Thanks,
> Julien
>
>