[Caml-list] BigArrays

[Caml-list] BigArrays

[David McClain]

I'm finally ready to consider moving up to the OCaml BigArrays in my NML. It
looks like a pretty good implementation. I still need to add special slicing
code that can extract arbitrary regions from arrays with strides different
from the parent array.

But one thing that stands out as a major weakness is the implementation of
memory mapped arrays. These currently do two objectionable things:

1. the entire file is mapped into memory -- these are potentially huge
files, bumping up to and sometimes exceeding 4 GB. Memory mapping needs to
use a sliding window of access in a manner transparent to the user. I often
default to 4 MB windows, but I do offer the possibility of a user settable
window size.

2. the file is mapped into memory starting at file offset zero. Most
datafiles use some amount of header information to describe the internal
layout of data. Data are mixed arrays of various machine word sizes, some
are double, others are float, and still others are unsigned character arrays
(images), all combined into the same large datafile. A file that contains
only one array of fixed wordsize is rather unusual.

Also curious about the limitation (??) to a maximum of 16 dimensions. While
I have never exceeded that limit, why place any limit?

David McClain
Senior Corporate Scientist
Avisere, Inc.

+1.520.390.7738 (USA)
david.mcclain@avisere.com



-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners