Re: [Caml-list] Array 4 MB size limit

[Jozef Kosoru <zyzstar@uid0.sk>]

On Sat, May 20, 2006 at 10:22:42 -0400, Brian Hurt wrote:
> >4G address space limit is off-topic here. Please note that the OCaml
> >array size limit is 4M not 4G.
> 
> It is entirely on-topic.  How much headroom do you think you have?
> 
> Let's perform an experiment here.  Let's assume that Xavier decides you're 
> right, and releases a new version of Ocaml with removes the 4M array 
> limit.  How big of an array will you be able to create then?  More than 
> 4M, yes- but how much more?
> 
> The hardware imposes a hard and fast 4G limit. [...] To make life
> simple, let's say that everything except the huge array will fit in
> the upper 2G, leaving 2G for the huge array.
> 
> Now if it's a huge array of integers, each element only takes up 4
> bytes, which means we can fit 512M integers in this array. [...] Wow,
> we could make an array 128 times as big as we can now! Except the
> proper way to look at this is that it's a mean 7 bits larger.  And
> that's best case.
> 
> If we changed it to an array of floats, each float now takes 8 bytes, and 
> that 2G can only hold 256M floats.  Our 128x improvement has now dropped 
> to a 64x improvement- and we're starting to get a bad feeling about where 
> this is going.
> 
> If we start trying to hold 3-dimensional pointers, that's 3 floats-
> and our 64x improvement has now dropped to about a 21x improvement.
> [...] making our 21x improvement (42x improvement) a mere 16x (25x)
> improvement.
> 
> Now, what happens if we try to hold a triangle of three points (i.e.
> for 3D polygon mapping)? [...] 2G can now hold about 19M unique
> polygons- a mere 5x advantage.

Thank you for elaborating on this issue and providing us some
interesting numbers. I think your results are valid. So it is perfectly
possible to find examples in a practice where there is a need to create
arrays from 5x to 128x bigger than OCaml allows. And that is the
problem.
 
> [...]
> 
> >I disagree. Actually I think an opposite. A suggestion to move to
> >64-bit is just patching the symptoms of the real problem - a design
> >flaw in the OCaml programming language. And this issue will not go
> >away on 64-bits.  Just like 22 bits for the array size is a
> >completely artificial limit on 32-bit platforms, (N - 10) imposes
> >once again such an unlogical 54 bits limit on 64-bit computers. And I
> >don't think "it makes complete sense on a 64-bit architecture".
> 
> By this logic, Ocaml should be able to support array sizes of 10^100th
> on 16-bit platforms.  Why is Ocaml imposing stupid artificial limits?

No. An above mentioned logic doesn't imply that. It should be 2^16 on
16-bits of course.
 
> I note with humor that the Java programming language a) defined ints
> to be 32 bits, and b) made the index type for arrays, vectors, and
> many other data structures ints and not longs.  When you move to 64
> bits, it's Java, much more so than Ocaml, that hits artificial and low
> limits on array size.  Ocaml will be able to have arrays millions of
> times larger than the largest possible array in Java.

Yeah, but we don't expect a lot of Java fans on this mailing list anyhow
;)
 
> >If you create OCaml programs for yourself then maybe you can switch
> >to 64-bits overnight (and perpahs you already did so). But if you
> >need to distribute your application to customers/users then it's much
> >more complicated. And explaining them that they should upgrade all
> >workstations because your software has various 4M limitations here
> >and there sounds like a bad joke, doesn't it?
> 
> You call them 32-bit limits, which they are.  I admit that this is a
> problem- but it's not one created by Ocaml.

It's been created by OCaml. CPU architecture is not obliged to have word
wide enough to allow programming languages to store some meta data along
with array indexes. That was OCaml "invention". An argument that 32-bit
is a crappy platform doesn't really hold. As Xavier acknowledged - OCaml
was not designed with 32-bit CPUs in mind; and that's the thing.
 
> >32-bit x86 platform is still good enough for many purposes and its
> >2^32 limit is acceptable in most cases while 2^22 is not.
> 
> Except that you're here comparing apples to oranges again.  The 2^22
> limit is in *words*, while the 2^32 limit is in *bytes*.  So it's
> really more of a 2^24 vr.s 2^31 comparison.

My point was that 2^32 (or 2^31 if you want) is usually enough to count
both apples and oranges whereas 2^22 is barely enough to count apple
trees.

But let's close this thread - I think we have discussed the problem into
great details and the final conclusion might be that consensus is not
possible. :)

Jozef
 
-- 
jozef kosoru
http://zyzstar.kosoru.com