Re: [Caml-list] unmarshaling large data from string on 32 bits

[Alain Frisch <alain.frisch@lexifi.com>]

Hello Fabrice,

While it's certainly interesting to extend the size of strings (and it 
would solve the original problem), it would not address the problem 
related to unmarshaling large data on 32-bit systems.


Alain

On 02/05/2015 01:22 PM, Fabrice Le Fessant wrote:
> Hi,
>
>    A long time ago, we had a patch on 32 bit systems to transparently
> extend the size of strings and arrays over the 16 MB limit:
>
> http://www.ocamlpro.com/blog/2011/05/06/longval.html
>
> At the time, there was little interest in this patch. If I remember
> well, it was quite small, we could probably include it in the next
> version of OCPWin32, if you think having it under Windows would help
> you.
>
> --Fabrice
>
>
> On Thu, Feb 5, 2015 at 11:50 AM, Alain Frisch <alain.frisch@lexifi.com> wrote:
>> On 02/05/2015 10:58 AM, Pierre-Marie Pédrot wrote:
>>>
>>> On 05/02/2015 09:56, Alain Frisch wrote:
>>>>
>>>> One possible work-around is to use an alternative implementation of the
>>>> demarshaler (there is such a pure OCaml implementation in Frama-C).
>>>> Another is to avoid the generic marshaling, either by writing a manual
>>>> version for your specific data type or by generating it from your type
>>>> definitions (à la bin-prot, I assume).
>>>
>>>
>>> Both workarounds would not work. Indeed, we use closure marshalling in
>>> Coq, which is not supported by the two proposed implementations.
>>
>>
>> Waow, closure marshaling across processes: you live dangerously :-)  I hope
>> you know precisely which values go into the closures or not (exception
>> slots, global mutable data structures, etc)...
>>
>>
>>> (Plus, that would be so slow I do not even want to think about it.)
>>
>>
>> I'm not so sure.  Serialization/deserialization routines specialized for
>> specific data types avoid some runtime checks required by the generic
>> functions (for the block tags and sizes, for detecting possible sharing
>> everywhere) and can use more specialized (and thus more compact) data
>> representation.
>>
>> We have had that problem at LexiFi, where we used to rely on the generic
>> marshaling for exchanges messages between processes (hence the ticket I
>> referred to).  We finally decided to write (manually) specialized functions
>> for our type of messages (no closures, no sharing), and the performance
>> results where slower but reasonable compared to the generic marshaling
>> (without putting too much engineering effort into it).  Anyway:
>>
>>   - our previous workaround was to trigger Gc.compact explicitly for big
>> messages, which was much worse, of course;
>>
>>   - it's clear that the OCaml implementation of the generic demarshaler would
>> be slower;
>>
>>   - since this only impacts 32-bit systems, nobody seems motivated enough to
>> put energy into fixing the core issue.
>>
>>
>> Alain
>>
>>
>> --
>> Caml-list mailing list.  Subscription management and archives:
>> https://sympa.inria.fr/sympa/arc/caml-list
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>
>