[Caml-list] Closures serialization and hash.

[Caml-list] Closures serialization and hash.

[Christophe Raffalli]

[-- Attachment #1: Type: text/plain, Size: 2075 bytes --]


Hello,

The usual limitation of serialization and hashing of closures is sometimes painful, especially with
the current temptation of parallelisation. I am involved in two projects/sofwares that are impacted
by this: bindlib[1] (where data structure using bound variables uses closures) and Patoline[2]
(where documents using animation and compiled with the "Bin" driver contain closures).

There is a simple way out of this problem, at least for libraries where the function pointer
in closure can be predicted enough :

  - use a table associating function "names" (position in the .cmo, or the lambda-tree or anything
portable) to source code adresses. The function name should be portable across architecture and
distinct binary using common librairies.

 - (1) fill this table by calling a function "register_code_pointer : ('a -> 'b) -> unit"

 - (2) or even better offer a linking option to register all closures from some compilation unit.

With this, serialisation and hashing functions could use those "names" instead of addresses. Thus we
would have reproducible hash and portable serialization for registered functions in closure.

I will probably try this soon by adding custom version of Hash and Marshal to bindlib using (1) ...
But support from real OCaml giving (2) would be much better.

So my question is : are you ennoyed by this problem, and, if yes, would you be happy with the above
solutions.

Cheers,
Christophe

[1] http://www.lama.univ-savoie.fr/~raffalli/bindlib
[2] http://www.patoline.com

-- 
Christophe Raffalli
Universite de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tel: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christophe.Raffalli@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature. The public key is
stored on www.keyserver.net
---------------------------------------------




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: [Caml-list] Closures serialization and hash.

[Stéphane Glondu]

Le 15/04/2013 11:16, Christophe Raffalli a écrit :
>   - use a table associating function "names" (position in the .cmo, or the lambda-tree or anything
> portable) to source code adresses. The function name should be portable across architecture and
> distinct binary using common librairies.
> 
>  - (1) fill this table by calling a function "register_code_pointer : ('a -> 'b) -> unit"

Instead of letting the compiler guessing a portable name, I'd rather
give explicit names to code pointers. Like Callback.register in the FFI.
You'd also get the possibility to maintain ABI stability.

>  - (2) or even better offer a linking option to register all closures from some compilation unit.

I'm not convinced this is better. I don't think having the possibility
to serialize /any/ closure is a good thing. IMHO, serialized functions
are like exported functions in modules. Their stability should be
guaranteeable at ABI level, which is not something you want for everything!


Cheers,

-- 
Stéphane

Re: [Caml-list] Closures serialization and hash.

[Gerd Stolpmann]

Am Montag, den 15.04.2013, 11:16 +0200 schrieb Christophe Raffalli:
> Hello,
> 
> The usual limitation of serialization and hashing of closures is sometimes painful, especially with
> the current temptation of parallelisation. I am involved in two projects/sofwares that are impacted
> by this: bindlib[1] (where data structure using bound variables uses closures) and Patoline[2]
> (where documents using animation and compiled with the "Bin" driver contain closures).
> 
> There is a simple way out of this problem, at least for libraries where the function pointer
> in closure can be predicted enough :
> 
>   - use a table associating function "names" (position in the .cmo, or the lambda-tree or anything
> portable) to source code adresses. The function name should be portable across architecture and
> distinct binary using common librairies.
> 
>  - (1) fill this table by calling a function "register_code_pointer : ('a -> 'b) -> unit"
> 
>  - (2) or even better offer a linking option to register all closures from some compilation unit.
> 
> With this, serialisation and hashing functions could use those "names" instead of addresses. Thus we
> would have reproducible hash and portable serialization for registered functions in closure.
> 
> I will probably try this soon by adding custom version of Hash and Marshal to bindlib using (1) ...
> But support from real OCaml giving (2) would be much better.
> 
> So my question is : are you ennoyed by this problem, and, if yes, would you be happy with the above
> solutions.

For the Plasma map/reduce platform, a similar problem arises. It starts
the same executable on several computers, and it is required to run
functions on remote instances. I've introduced so-called "registered
functions" to address this problem:

http://plasma.camlcity.org/plasma/dl/plasma-0.6.2/doc/html/Plasmamr_toolkit.html#2_Registeredfunctions

I've added some camlp4 support to make the registration less painful, so
you can simply do

let f = <:rfun< variables -> body >>

This concept also distinguishes between local arguments and remote
arguments. The local arguments are normal function arguments, and are
not marshalled, as if you did

let f localarg = <:rfun ...>>

(only that this form would not run any registration code). The remote
arguments are the marshalled ones.

Of course, there is still the problem that we cannot ensure that the
registration of the function is done before its use, simply because we
don't know whether the registration is done in a toplevel module (I
guess this is why you'd prefer (2) ).

Gerd





> 
> Cheers,
> Christophe
> 
> [1] http://www.lama.univ-savoie.fr/~raffalli/bindlib
> [2] http://www.patoline.com
> 

-- 
------------------------------------------------------------
Gerd Stolpmann, Bad Nauheimer Str.3, 64289 Darmstadt,Germany 
gerd@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------