Re: [Caml-list] Array 4 MB size limit

[j h woodyatt <jhw@conjury.org>]

On May 20, 2006, at 7:22 AM, Brian Hurt wrote:
>
> Consider the classic programming pattern of reading the entire file  
> into a list (or array) of strings.  [...] boom- welcome to the 4G  
> limit.  Heck, you have to do special incantations just to open and  
> read a file that big, let only trying to store the whole thing in  
> memory.  This is why I claim that once hard disks start getting  
> larger than the address space, the address space needs to increase.

Sounds like it's an *anti*-pattern to me.  The offset parameter to  
mmap(2) is there for a reason.

There is, however, another interesting point I hope the Caml team at  
INRIA keeps in mind.  I hear rumors that 64-bit processes on  
[redacted] may not have access to all the application support  
libraries.  Some of them, e.g. [redacted] and [redacted], may only be  
available to 32-bit processes-- for technical reasons, not business  
reasons.  The engineering resources to make the libraries available  
to 64-bit processes are there, but the technical case for making 64- 
bit versions of the libraries is apparently losing.  At least for the  
time being.  I would be unsurprised if technical decisions like this  
are happening all over the application services programming world.

What this tells me is that 64-bit hardware and 64-bit operating  
systems will very likely continue to have 32-bit processes running on  
them for some time to come.  We will be living with 32-bit address  
spaces for many years after everyone gets 64-bit operating systems  
and 64-bit desktop machines.

All that said, I've got little sympathy for the critics on this  
issue.  The 4 MB array size limit is a fact of OCaml life, but  
there's a reasonable workaround that will get most application  
programmers around it, i.e. careful use of Bigarray.  The 4G size  
limit is another story altogether, and it will plague everyone  
equally-- not just the OCaml world.


—
j h woodyatt <jhw@conjury.org>