Re: [Caml-list] non-square big arrays

[Brian Hurt <bhurt@janestcapital.com>]

Hal Daume III wrote:

>I have code that, right now, uses "int array array" to store lots of
>things, but in some cases I run out of indices in the outer array (the
>inner arrays are always fine, but I hit the max array length limit on
>the outside).
>
>I would be happy to switch over to bigarrays, but my understanding is
>that 2D bigarrays have to be square.  I cannot possible have square
>arrays (it would take too much memory).  Is there an obvious way around
>this that I'm missing?
>
>  
>
I don't think so.  2D  bigarrays are basically C pointer arithmetic 
games.  My recommendations would be, in order:

1) switch to 64 bit if at all humanly possible, at which point the 
problem goes away.  Also, you're likely to be hitting memory limits in 
any case.  If each subarray takes more than a few hundred bytes of 
space, you'll be blowing 32-bit address space.

2) Make a new "hugearray" module which creates a virutal array that can 
be larger than the limit of simple arrays by storing it in an array of 
arrays.  This isn't that hard to write. Keep the top level array small 
(1024 entries should be large enough) so it lives in cache, and the 
implementation probably wouldn't be signifigantly slower than a single 
flat array.

Brian