non-square big arrays

non-square big arrays

[Hal Daume III]

I have code that, right now, uses "int array array" to store lots of
things, but in some cases I run out of indices in the outer array (the
inner arrays are always fine, but I hit the max array length limit on
the outside).

I would be happy to switch over to bigarrays, but my understanding is
that 2D bigarrays have to be square.  I cannot possible have square
arrays (it would take too much memory).  Is there an obvious way around
this that I'm missing?

-- 
 Hal Daume III --- me AT hal3 DOT name  |  http://hal3.name
 "Arrest this man, he talks in maths."  |  http://nlpers.blogspot.com

Re: [Caml-list] non-square big arrays

[Brian Hurt]

Hal Daume III wrote:

>I have code that, right now, uses "int array array" to store lots of
>things, but in some cases I run out of indices in the outer array (the
>inner arrays are always fine, but I hit the max array length limit on
>the outside).
>
>I would be happy to switch over to bigarrays, but my understanding is
>that 2D bigarrays have to be square.  I cannot possible have square
>arrays (it would take too much memory).  Is there an obvious way around
>this that I'm missing?
>
>  
>
I don't think so.  2D  bigarrays are basically C pointer arithmetic 
games.  My recommendations would be, in order:

1) switch to 64 bit if at all humanly possible, at which point the 
problem goes away.  Also, you're likely to be hitting memory limits in 
any case.  If each subarray takes more than a few hundred bytes of 
space, you'll be blowing 32-bit address space.

2) Make a new "hugearray" module which creates a virutal array that can 
be larger than the limit of simple arrays by storing it in an array of 
arrays.  This isn't that hard to write. Keep the top level array small 
(1024 entries should be large enough) so it lives in cache, and the 
implementation probably wouldn't be signifigantly slower than a single 
flat array.

Brian