From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=AWL autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from concorde.inria.fr (concorde.inria.fr [192.93.2.39]) by yquem.inria.fr (Postfix) with ESMTP id 41394BC69 for ; Tue, 6 Feb 2007 22:16:13 +0100 (CET) Received: from smtp.janestcapital.com (www.janestcapital.com [66.155.124.107]) by concorde.inria.fr (8.13.6/8.13.6) with ESMTP id l16LG5QP005292 for ; Tue, 6 Feb 2007 22:16:08 +0100 Received: from [192.168.250.117] [209.213.205.130] by janestcapital.com with ESMTP (SMTPD-9.10) id A01304A4; Tue, 06 Feb 2007 16:16:03 -0500 Message-ID: <45C8F013.5050509@janestcapital.com> Date: Tue, 06 Feb 2007 16:16:03 -0500 From: Brian Hurt User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax) X-Accept-Language: en-us, en MIME-Version: 1.0 To: me@hal3.name Cc: caml-list@yquem.inria.fr Subject: Re: [Caml-list] non-square big arrays References: <45C8EC31.6060300@cs.utah.edu> In-Reply-To: <45C8EC31.6060300@cs.utah.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Miltered: at concorde with ID 45C8F015.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; arrays:01 arrays:01 bigarrays:01 bigarrays:01 pointer:01 wrote:01 slower:01 caml-list:01 arithmetic:01 int:01 bytes:03 module:03 brian:04 brian:04 entries:05 Hal Daume III wrote: >I have code that, right now, uses "int array array" to store lots of >things, but in some cases I run out of indices in the outer array (the >inner arrays are always fine, but I hit the max array length limit on >the outside). > >I would be happy to switch over to bigarrays, but my understanding is >that 2D bigarrays have to be square. I cannot possible have square >arrays (it would take too much memory). Is there an obvious way around >this that I'm missing? > > > I don't think so. 2D bigarrays are basically C pointer arithmetic games. My recommendations would be, in order: 1) switch to 64 bit if at all humanly possible, at which point the problem goes away. Also, you're likely to be hitting memory limits in any case. If each subarray takes more than a few hundred bytes of space, you'll be blowing 32-bit address space. 2) Make a new "hugearray" module which creates a virutal array that can be larger than the limit of simple arrays by storing it in an array of arrays. This isn't that hard to write. Keep the top level array small (1024 entries should be large enough) so it lives in cache, and the implementation probably wouldn't be signifigantly slower than a single flat array. Brian