Re: [Caml-list] Array 4 MB size limit

[Frederick Akalin <akalin@akalin.cx>]

Xavier Leroy wrote:
>> I was greatly surprised when I found out there was such a low limit on
>> arrays.  Is there a reason for this?  Will this limit ever be increased?
>>     
>
> As Brian Hurt explained, this limit comes from the fact that heap
> object sizes are stored in N-10 bits, where N is the bit width of the
> processor (32 or 64).
>
> Historical digression: this representation decision was initially
> taken when designing Caml Light in 1989-1990.  At that time, even
> professional workstations ... Little did I know that the 32-bitters would
> survive so long.  Now, it's 2006, and 64-bit processors are becoming
> universally available, in desktop machines at least.  (I've been
> running an AMD64 PC at home since january 2005 with absolutely zero
> problems.)  So, no the data representations of OCaml are not going to
> change to lift the array size limit on 32-bit machines.
>
>   
I see.  This is a perfectly reasonable explanation.  I ended up just 
using Bigarrays of floats for now and converting from those to 3-d 
vectors on demand (what I need), but it would be nice to have a type 
that wraps around Array that can get around the 4M limit (using an array 
of arrays like someone suggested earlier).  It's sad, but I think 32-bit 
is going to be around for a while, and surely I can't be the only person 
to run up against this. :) Not that I mind writing such a library and 
releasing it.  I wonder if the Extlib guys would be interested...
> A better idea would be to determine exactly what data structure you need:
> which abstract operations, what performance requirements, etc.  C++
> and Lisp programmers tend to encode everything as arrays or lists,
> respectively, but quite often these are not the best data structure
> for the application of interest.
>   
I find your assertion surprising.  C++ and Common LISP are by no means 
lacking in standard data structures (and using bare arrays is 
discouraged in C++, as far as I know) and in my experience I haven't 
much seen C++ code that used arrays/vectors when not appropriate.

In any case, in my application (a raytracer) I am reading in lists of 
numbers from a file representing the points of a mesh and the triangles 
that make up the mesh (represented by a list of indices into the mesh 
list).  A dynamically resizable, reasonable scalable array seems like 
the best data structure to use.
>> Also, the fact that using lists crashes for the same data set is
>> surprising.  Is there a similar hard limit for lists, or would this be a
>> bug?  Should I post a test case?
>>     
>
> Depends on the platform you use.  In principle, Caml should report
> stack overflows cleanly, by throwing a Stack_overflow exception.
> However, this is hard to do in native code as it depends a lot on the
> processor and OS used.  So, some combinations (e.g. x86/Linux) will
> report stack overflows via an exception, and others will let the
> kernel generate a segfault.
>
> If you're getting the segfault under x86/Linux for instance, please
> post a test case on the bug tracking system.  It's high time that
> Damien shaves :-)
>
>   
Due to some confusion on my part (I didn't realize stderr was buffered 
in OCaml and needed to be flushed), I erroneously thought my program was 
crashing on building the list.  It was, in fact, a List.map lurking 
further along in my code.  I'm running PPC/OS X, which I'm guessing is 
one of those platforms that lets the kernel generate a segfault?  I 
tried it on Linux and it threw an exception as expected.  Damien's 
facial hair is safe. :)
> - Xavier Leroy
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>   
Frederick Akalin