Re: [Caml-list] unmarshaling large data from string on 32 bits

[Alain Frisch <alain.frisch@lexifi.com>]

Hello,

Be aware when using the generic demarshaling on 32 bit systems with 
large data (even when they fit in a string):  this will expand the heap 
(adding more pages to it) on every demarshaling, and unless you arrange 
so that the compacter runs often enough (calling manually Gc.compact for 
instance), you'll end up eating all the memory.

This is documented here:

http://caml.inria.fr/mantis/view.php?id=5813

One possible work-around is to use an alternative implementation of the 
demarshaler (there is such a pure OCaml implementation in Frama-C). 
Another is to avoid the generic marshaling, either by writing a manual 
version for your specific data type or by generating it from your type 
definitions (à la bin-prot, I assume).


Alain



On 02/02/2015 11:32 AM, Enrico Tassi wrote:
> Hello, I've just discovered that on 32 bits systems strings are
> limited to 16M.  I'm using strings as buffers holding data to
> be unmarshaled.  I could use another data structure, like a Buffer.t,
> but I see no API for unmarshaling from a Buffer.t.
>
> Is there another way? Is there code out there implementing that?
>
> Best,
>