RE: [Caml-list] Wish List for Large Mutable Objects

["Brandon J. Van Every" <vanevery@indiegamedesign.com>]

David McClain wrote:
>
> Something I would like to see appear in the OCaml libraries,
> and I don't have it yet myself, is the use of Copy-on-Write
> and Scatter-Gather applied to large mutable objects such as
> BigArrays.
>
> [and some other things in other posts]

OCaml is not a lazy language.  Is it reasonable to expect Bigarray to
perform lazy copies, under any permutation of complexity you have in
mind?

It seems like what you really want is to design the memory management of
an Operating System.  If your files are so huge that they don't fit in
main memory, why aren't you willing to use virtual memory?
Scatter-Gather DMA is a device driver level capability that's hardware
dependent.  I don't see why a high level language like OCaml should be
exposing that kind of functionality, and I'm not entirely sure if it
should be doing it under the hood either.  If your OS doesn't have the
kind of memory management you want, maybe you should modify an open
source OS, like Linux or BSD Unix, to do what you want?

You ask why Array1, Array2, Array3 should be special cases.  Well,
clearly because they're the most common, and you can perform access
optimizations for each of these common cases.  It seems that you are
only thinking of ***BIG*** arrays, i.e. your problems and nobody else's.
Lotsa people don't have your notion of 'big'.  Indeed, I don't
personally care about arrays being particularly big.  100MB would be
pretty darn big for what I do in game development right now.  I do care
about their contents being unboxed.  If someone wanted to rename
Bigarray to UnboxedArray, that would suit my own priorities just fine.

I don't understand the "starting from zero" complaint, with respect to
arbitrary file formats.  If the file format is arbitrarily structured,
it is not an array.  You will have to read it some other way.  Arrays
are, generally speaking, composed of uniform elements.  At least, that's
how all of us pedal-to-the-metal guys view them.  I suppose high level
language guys often define the word 'array' to mean anything they want,
like a list or a hash table or a map or whatever, but I don't think they
should.

I don't see why your Scientific notion of an 'infinite array' should be
a basic language interface.  What would be so difficult about building
your favorite array windowing scheme on top of the basic fixed length
components, and calling that a library?  Like 'InfiniteArray' or
something.  Then you'd write some access functions in some syntax you
like, it would behave the way you like, and for your problems you'd be
good.  I've done similar things to perform addressing on icosahedrons,
to try to regularize the mathematics of a tiling of it.  I don't bother
the user about it, my functions just do some computing to make it all
work under the hood.

I do wish Bigarray handled heterogeneous C structures.  Homogeneous
arrays impose some design and interop constraints.

Finally, I'm told that the "%" in the names of called functions in the
sources means that ocamlopt generates different, better code.  The C
routines are ignored, they're only used for ocamlc.


Cheers,                         www.indiegamedesign.com
Brandon Van Every               Seattle, WA

20% of the world is real.
80% is gobbledygook we make up inside our own heads.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners