Re: [Caml-list] unmarshaling large data from string on 32 bits

[Fabrice Le Fessant <Fabrice.Le_fessant@inria.fr>]

Hi,

  A long time ago, we had a patch on 32 bit systems to transparently
extend the size of strings and arrays over the 16 MB limit:

http://www.ocamlpro.com/blog/2011/05/06/longval.html

At the time, there was little interest in this patch. If I remember
well, it was quite small, we could probably include it in the next
version of OCPWin32, if you think having it under Windows would help
you.

--Fabrice


On Thu, Feb 5, 2015 at 11:50 AM, Alain Frisch <alain.frisch@lexifi.com> wrote:
> On 02/05/2015 10:58 AM, Pierre-Marie Pédrot wrote:
>>
>> On 05/02/2015 09:56, Alain Frisch wrote:
>>>
>>> One possible work-around is to use an alternative implementation of the
>>> demarshaler (there is such a pure OCaml implementation in Frama-C).
>>> Another is to avoid the generic marshaling, either by writing a manual
>>> version for your specific data type or by generating it from your type
>>> definitions (à la bin-prot, I assume).
>>
>>
>> Both workarounds would not work. Indeed, we use closure marshalling in
>> Coq, which is not supported by the two proposed implementations.
>
>
> Waow, closure marshaling across processes: you live dangerously :-)  I hope
> you know precisely which values go into the closures or not (exception
> slots, global mutable data structures, etc)...
>
>
>> (Plus, that would be so slow I do not even want to think about it.)
>
>
> I'm not so sure.  Serialization/deserialization routines specialized for
> specific data types avoid some runtime checks required by the generic
> functions (for the block tags and sizes, for detecting possible sharing
> everywhere) and can use more specialized (and thus more compact) data
> representation.
>
> We have had that problem at LexiFi, where we used to rely on the generic
> marshaling for exchanges messages between processes (hence the ticket I
> referred to).  We finally decided to write (manually) specialized functions
> for our type of messages (no closures, no sharing), and the performance
> results where slower but reasonable compared to the generic marshaling
> (without putting too much engineering effort into it).  Anyway:
>
>  - our previous workaround was to trigger Gc.compact explicitly for big
> messages, which was much worse, of course;
>
>  - it's clear that the OCaml implementation of the generic demarshaler would
> be slower;
>
>  - since this only impacts 32-bit systems, nobody seems motivated enough to
> put energy into fixing the core issue.
>
>
> Alain
>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa.inria.fr/sympa/arc/caml-list
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs



-- 
Fabrice LE FESSANT
Chercheur en Informatique
INRIA Paris Rocquencourt -- OCamlPro
Programming Languages and Distributed Systems