Re: [Caml-list] Wish List for Large Mutable Objects

[Ville-Pertti Keinonen <will@exomi.com>]

Brandon J. Van Every wrote:

>loop optimizations for 3d graphics.  That said, Scatter-Gather DMA is
>generally a property of a memory controller, i.e. a chipset on a
>motherboard.  I seriously doubt you have user mode access to such memory
>controllers.  If you do, point me at the API for it.  I'm happy to stand
>corrected, but as far as I know, scatter-gather DMA is kernel mode stuff
>on all common architectures.
>  
>
Often things like the readv(2)/writev(2) interface are referred to as 
"scatter-gather".  It just means I/O on regions of memory that aren't 
contiguous in a single operation.

I'm not sure what David McClain is referring to - but I think it's the 
ability for an "array" to provide another level of virtualization so 
that the underlying data needn't be contiguous in the address space of 
the process.

That seems a bit excessive - a more limited part of what he's suggesting 
could be more reasonable - being able to map a part of a file, from an 
arbitrary offset, as a Bigarray could be useful.  Even this would 
require some separation of storage management from the actual "Bigarray 
header", since operating systems require the underlying mappings to be 
page aligned.  I suspect this could be as simple as passing a 
page-truncated offset to mmap(2), adding the remaining offset to the 
returned address and page-truncating the address passed to munmap(2).

This doesn't address the problem with most CPUs requiring the actual 
objects to be aligned, for which adding an "offset" between the 
beginning of the mapping and the beginning of the visible array isn't 
sufficient if the mapped file doesn't align things appropriately for the 
CPU (for arbitrary file formats, there's also endianness issues to 
consider).

Using subarrays instead of offsetted memory mappings protects against 
this, and makes offsetted mappings "unnecessary" altogether, but 
obviously if the file is big enough, the waste of address space due to 
the extra mappings can be significant on 32-bit systems...which as I 
understand was part of the original problem.

Typing this on an Athlon 64 and sitting next to an Alpha, such things 
seem like legacy issues to me, especially since OCaml supports both 
architectures natively.

The main issue I have with the suggestions regarding turning Bigarrays 
into higher-level abstractions altogether is that it would make them 
considerably less efficient.  The higher-level abstractions can always 
be implemented as layers on top of the current abstractions, which in my 
opinion is the right approach.

Of course it isn't my decision in any way.

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners