[Caml-list] using BigArray to mmap huge binary file

[Caml-list] using BigArray to mmap huge binary file

[Dário Abdulrehman]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

Hi,

I have a huge binary data file that I would like to memory map for intensive
reading (no writing) with the best possible performance.
I read some discussions about using BigArrays to achieve this in Ocaml.  I
would like to know how this solution compares with the C language mmap.
Please bear in mind I am a total newbie to Ocaml but I have good Common Lisp
experience.

Thanks.

[-- Attachment #2: Type: text/html, Size: 419 bytes --]

Re: [Caml-list] using BigArray to mmap huge binary file

[Richard Jones]

On Thu, Nov 09, 2006 at 06:30:50PM +0000, Dário Abdulrehman wrote:
> I have a huge binary data file that I would like to memory map for intensive
> reading (no writing) with the best possible performance.
> I read some discussions about using BigArrays to achieve this in Ocaml.  I
> would like to know how this solution compares with the C language mmap.
> Please bear in mind I am a total newbie to Ocaml but I have good Common Lisp
> experience.

We use Bigarray to map geographical data.  It works.  As far as I'm
aware ArrayX.map_file is really just a wrapper around mmap, and read
accesses are just indirections on a pointer.

Depends a lot on what format your data is in.  Ours is stored as a
huge array of 16-bit signed ints and Bigarray supports that naturally.
If your data was in an unsupported format, or if it wasn't strictly an
array, you might end up doing contortions.

Rich.

-- 
Richard Jones, CTO Merjis Ltd.
Merjis - web marketing and technology - http://merjis.com
Internet Marketing and AdWords courses - http://merjis.com/courses - NEW!
Merjis blog - http://blog.merjis.com - NEW!

Re: [Caml-list] using BigArray to mmap huge binary file

[malc]

On Thu, 9 Nov 2006, Richard Jones wrote:

> On Thu, Nov 09, 2006 at 06:30:50PM +0000, D?rio Abdulrehman wrote:
>> I have a huge binary data file that I would like to memory map for intensive
>> reading (no writing) with the best possible performance.
>> I read some discussions about using BigArrays to achieve this in Ocaml.  I
>> would like to know how this solution compares with the C language mmap.
>> Please bear in mind I am a total newbie to Ocaml but I have good Common Lisp
>> experience.
>
> We use Bigarray to map geographical data.  It works.  As far as I'm
> aware ArrayX.map_file is really just a wrapper around mmap, and read
> accesses are just indirections on a pointer.

I beg to differ, to the best of my knowlege every reference to bigarray
is bounds checked, even with -unsafe. So, while map_file does indeed look
like a wrapper around mmap, read/write is a bit more than an indirected
memory fetch.

--
vale