Re: [Caml-list] unmarshaling large data from string on 32 bits

[Alain Frisch <alain.frisch@lexifi.com>]

On 02/05/2015 10:58 AM, Pierre-Marie Pédrot wrote:
> On 05/02/2015 09:56, Alain Frisch wrote:
>> One possible work-around is to use an alternative implementation of the
>> demarshaler (there is such a pure OCaml implementation in Frama-C).
>> Another is to avoid the generic marshaling, either by writing a manual
>> version for your specific data type or by generating it from your type
>> definitions (à la bin-prot, I assume).
>
> Both workarounds would not work. Indeed, we use closure marshalling in
> Coq, which is not supported by the two proposed implementations.

Waow, closure marshaling across processes: you live dangerously :-)  I 
hope you know precisely which values go into the closures or not 
(exception slots, global mutable data structures, etc)...


> (Plus, that would be so slow I do not even want to think about it.)

I'm not so sure.  Serialization/deserialization routines specialized for 
specific data types avoid some runtime checks required by the generic 
functions (for the block tags and sizes, for detecting possible sharing 
everywhere) and can use more specialized (and thus more compact) data 
representation.

We have had that problem at LexiFi, where we used to rely on the generic 
marshaling for exchanges messages between processes (hence the ticket I 
referred to).  We finally decided to write (manually) specialized 
functions for our type of messages (no closures, no sharing), and the 
performance results where slower but reasonable compared to the generic 
marshaling (without putting too much engineering effort into it).  Anyway:

  - our previous workaround was to trigger Gc.compact explicitly for big 
messages, which was much worse, of course;

  - it's clear that the OCaml implementation of the generic demarshaler 
would be slower;

  - since this only impacts 32-bit systems, nobody seems motivated 
enough to put energy into fixing the core issue.


Alain