Re: [Caml-list] Storing ocaml values outside ocaml's heap

[William Le Ferrand <William.Le-Ferrand@polytechnique.edu>]

[-- Attachment #1: Type: text/plain, Size: 2990 bytes --]

Hi Anders!

How're you doing ?

Thanks for the suggestion. We actually have a solution in production right
now (www.besport.com) that relies on netshm. Unfortunately, the
serialization / deserialization is killing the performance : about 200ms
for a full request against 7ms for a simple Hashtbl lookup on our hardware.

ocaml-everlasting did solve this performance bottleneck but we get
unacceptable segmentation faults, and before debugging this library I'd
rather learn about other approaches :)

Best regards,

William


On Thu, Dec 8, 2011 at 12:49 AM, Anders Fugmann <anders@fugmann.net> wrote:

> Hi William,
>
> Have you tried netshm from ocamlnet library? It seems that it would fit
> your requirements by placing memory outside the reach of the garbage
> collector.
>
> Using netshm would require a lot of serialization / deserilization and
> memory copying, which might not be that efficient if you access the cache a
> lot.
>
> Regards
> Anders Fugmann
>
>
>
>
> On 12/08/2011 05:35 AM, William Le Ferrand wrote:
>
>> Dear list,
>>
>> We are building a cache in ocaml and we're wondering if it would make
>> sense to store ocaml values outside the reach of the gc. (gc on a 20GB
>> cache hangs the process for a second or so).
>>
>> To run some experiments, we wrote a small library
>> (https://github.com/besport/**ocaml-everlasting<https://github.com/besport/ocaml-everlasting>)
>> that exposes two
>> functions, get and set.
>>
>> When inserting a value, we copy recursively the blocs outside of the
>> reach of the gc (and put the resulting value in some C array). When
>> getting the value, we simply pass the pointer to the copied value to the
>> ocaml code (the structure is still coherent and the value is directly
>> usable). We also wrote an "update" function that compare a new value
>> with the existing value in cache, to avoid unnecessary memory
>> allocation/deallocation.
>>
>> It does not seems very stable though, but I don't know if it is a bug in
>> the update function or simply because this approach is not reasonable.
>> Do you have any thoughts? Is there any clever way to build a large cache
>> in an ocaml app ?
>>
>> Thanks in advance for any tips!
>>
>> Best
>>
>> William
>>
>>
>> --
>> William Le Ferrand
>>
>> Mobile : (+1) (415) 683-1484
>> Web : http://williamleferrand.**github.com/<http://williamleferrand.github.com/>
>> <http://www.linkedin.com/in/**williamleferrand<http://www.linkedin.com/in/williamleferrand>
>> >
>>
>>
>
> --
> Caml-list mailing list.  Subscription management and archives:
> https://sympa-roc.inria.fr/**wws/info/caml-list<https://sympa-roc.inria.fr/wws/info/caml-list>
> Beginner's list: http://groups.yahoo.com/group/**ocaml_beginners<http://groups.yahoo.com/group/ocaml_beginners>
> Bug reports: http://caml.inria.fr/bin/caml-**bugs<http://caml.inria.fr/bin/caml-bugs>
>
>


-- 
William Le Ferrand

Mobile : (+1) (415) 683-1484
Web : http://williamleferrand.github.com/
<http://www.linkedin.com/in/williamleferrand>

[-- Attachment #2: Type: text/html, Size: 4468 bytes --]