ocamlc.opt tries to allocate 34 GB of RAM ... and fails

ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones]

We had a bug reported ages ago where ocamlc.opt would try to allocate
a ridiculous amount of RAM (34289692672 bytes, or about 34 GB) and
fail.  Another very similar bug report[1] has just come along, with a
nice small reproducer:

------------------------------------------------------------ log.ml
open Camlp4.PreCast;
open Syntax;


value insert_mname_mline _loc f = <:expr<Log.$lid:f$ $str:Loc.file_name  _loc$
$`int:Loc.start_line _loc$>>;

EXTEND Gram
  GLOBAL: expr;

    expr:
      [
        [ "Log"; "."; f = [ `LIDENT ("e"|"d"|"w" as f) -> f] ->
insert_mname_mline _loc f ]
      ];
  END;
----------------------------------------------------------------------

Running:

  ocamlc.opt -c -I +camlp4 -pp camlp4rf log.ml

_sometimes_ fails with an out of memory error, eg:

  for i in `seq 1 100`; do ocamlc.opt -c -I +camlp4 -pp camlp4rf log.ml; done

For me it fails about 1 time in 10.

There's some evidence this may be a Fedora-specific problem, maybe
some problem in the kernel or glibc or how we build OCaml, however I
have no real idea so I'm wondering if anyone can try reproducing it.

Rich.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=445545

-- 
Richard Jones
Red Hat

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Romain Beauxis]

Isn't it this bug:

  http://caml.inria.fr/mantis/view.php?id=4448





Romain

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones]

On Wed, May 07, 2008 at 07:35:44PM +0200, Romain Beauxis wrote:
> Isn't it this bug:
> 
>   http://caml.inria.fr/mantis/view.php?id=4448

It is suggested in that bug report that turning off address space
randomization should make the problem go away.  But I have turned this
off, and there is no difference.

Also I checked the source to our caml_aligned_mmap function and it is
essentially the same as Markus Mottl's version in his comment in that
thread (this has been reported in 3.10.1 & 3.10.2).  It even includes
a reference to that bug, indicating that the version we are using
includes this patch.

I'd still like to know if anyone can reproduce this case.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Markus Mottl]

On Wed, May 7, 2008 at 2:34 PM, Richard Jones <rich@annexia.org> wrote:
>  Also I checked the source to our caml_aligned_mmap function and it is
>  essentially the same as Markus Mottl's version in his comment in that
>  thread (this has been reported in 3.10.1 & 3.10.2).  It even includes
>  a reference to that bug, indicating that the version we are using
>  includes this patch.

Note that the patch does not guarantee anything.  The mmap system call
only considers the passed address as a hint, there is no guarantee
that it will actually use it.  This may be highly kernel-dependent.

Note, too, that the upcoming OCaml-release will (probably) feature
improved page table handling using sparse representations, thus
solving this problem generally.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones]

On Wed, May 07, 2008 at 02:46:49PM -0400, Markus Mottl wrote:
> On Wed, May 7, 2008 at 2:34 PM, Richard Jones <rich@annexia.org> wrote:
> >  Also I checked the source to our caml_aligned_mmap function and it is
> >  essentially the same as Markus Mottl's version in his comment in that
> >  thread (this has been reported in 3.10.1 & 3.10.2).  It even includes
> >  a reference to that bug, indicating that the version we are using
> >  includes this patch.
> 
> Note that the patch does not guarantee anything.  The mmap system call
> only considers the passed address as a hint, there is no guarantee
> that it will actually use it.  This may be highly kernel-dependent.

OK, but is there any case where ocamlc.opt should try to allocate
34 _gigabytes_ (in the 2nd [length] param of mmap(2)).

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Markus Mottl]

On Wed, May 7, 2008 at 2:49 PM, Richard Jones <rich@annexia.org> wrote:
>  OK, but is there any case where ocamlc.opt should try to allocate
>  34 _gigabytes_ (in the 2nd [length] param of mmap(2)).

Hm, this seems overly excessive.  I haven't been able to reproduce the
problem on our Fedora 5 boxes (32 and 64bit) using any (including
older) OCaml-compiler.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Brian Hurt]

[-- Attachment #1: Type: text/plain, Size: 615 bytes --]

Markus Mottl wrote:

>On Wed, May 7, 2008 at 2:49 PM, Richard Jones <rich@annexia.org> wrote:
>  
>
>> OK, but is there any case where ocamlc.opt should try to allocate
>> 34 _gigabytes_ (in the 2nd [length] param of mmap(2)).
>>    
>>
>
>Hm, this seems overly excessive.  I haven't been able to reproduce the
>problem on our Fedora 5 boxes (32 and 64bit) using any (including
>older) OCaml-compiler.
>
>Regards,
>Markus
>
>  
>
This is possible- if the ocaml heap was spread out over 2^48 bytes, then 
the old implementation would want to allocate 32G or more.  I'd be 
surprised, but this is possible...

Brian


[-- Attachment #2: Type: text/html, Size: 1145 bytes --]

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones]

On Wed, May 07, 2008 at 03:24:07PM -0400, Markus Mottl wrote:
> On Wed, May 7, 2008 at 2:49 PM, Richard Jones <rich@annexia.org> wrote:
> >  OK, but is there any case where ocamlc.opt should try to allocate
> >  34 _gigabytes_ (in the 2nd [length] param of mmap(2)).
> 
> Hm, this seems overly excessive.  I haven't been able to reproduce the
> problem on our Fedora 5 boxes (32 and 64bit) using any (including
> older) OCaml-compiler.

It only seems to happen in Fedora 9, where coincidentally we switched
to using gcc 4.3.  Anyway, I'm still looking at the code ...

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones]

OK I've found out where it's going wrong and I have a theory, ill-informed
probably, about the problem.

Firstly when I run the compiler with OCAMLRUNPARAM=v=0x1ff, just
before the error I see:

  Growing heap to 4320k bytes
  Growing page table to 34359705221 entries
  No room for growing page table
  Fatal error: out of memory.

and running it under gdb seems to indicate that the problem is a
massive malloc() request, the second call to malloc in the function
caml_add_to_heap.

Now my uninformed theory: The preceeding mmap (from caml_aligned_mmap)
returned a very high memory address:

  mmap(0x722020, 495648, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7ffff7f83000

and the runtime is trying to construct a page table (bitmap, right?)
to track both very low and very high addresses.  That's failing,
badly.

Does this sound right?

Unfortunately I have no idea how to solve this, certainly it seems
there is no simple fix ...  (I tried to set vm.overcommit_memory
policy, but that doesn't work because the page table is initialized
right after allocation).

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Markus Mottl]

On Thu, May 8, 2008 at 6:14 AM, Richard Jones <rich@annexia.org> wrote:
>   Growing heap to 4320k bytes
>   Growing page table to 34359705221 entries
>   No room for growing page table
>   Fatal error: out of memory.

Yeah, that's an indication that you've hit the mentioned problem
(addresses of allocated memory too far apart).

>  Unfortunately I have no idea how to solve this, certainly it seems
>  there is no simple fix ...  (I tried to set vm.overcommit_memory
>  policy, but that doesn't work because the page table is initialized
>  right after allocation).

If my mmap "fix" (not guaranteed to work with all kernels) is in the
executing runtime, then, I'm afraid, there is little you can do other
than switching to a CVS-version of the upcoming OCaml-release.  Xavier
has implemented a completely new page table representation there,
which should solve this problem once and for all.

Regards,
Markus

-- 
Markus Mottl http://www.ocaml.info markus.mottl@gmail.com

Re: [Caml-list] ocamlc.opt tries to allocate 34 GB of RAM ... and fails

[Richard Jones]

On Thu, May 08, 2008 at 09:40:12AM -0400, Markus Mottl wrote:
> On Thu, May 8, 2008 at 6:14 AM, Richard Jones <rich@annexia.org> wrote:
> >   Growing heap to 4320k bytes
> >   Growing page table to 34359705221 entries
> >   No room for growing page table
> >   Fatal error: out of memory.
> 
> Yeah, that's an indication that you've hit the mentioned problem
> (addresses of allocated memory too far apart).

Yes, we came a similar conclusion.

However there is a MAP_32BIT flag that you can add to mmap which makes
it return low memory addresses.  There are two problems with this,
namely (a) the flag is only available on x86-64 (and we have the same
problem on ppc64), and (b) it limits the whole heap to 1 or 2 GB.
Nevertheless this is the solution we have gone with for now.

  http://lxr.linux.no/linux/include/asm-x86_64/mman.h#L6
  https://bugzilla.redhat.com/attachment.cgi?id=304848
  https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9

Hopefully OCaml 3.11 will arrive before Fedora 10 is released, in
which case we'll just upgrade.

> >  Unfortunately I have no idea how to solve this, certainly it seems
> >  there is no simple fix ...  (I tried to set vm.overcommit_memory
> >  policy, but that doesn't work because the page table is initialized
> >  right after allocation).
> 
> If my mmap "fix" (not guaranteed to work with all kernels) is in the
> executing runtime, then, I'm afraid, there is little you can do other
> than switching to a CVS-version of the upcoming OCaml-release.  Xavier
> has implemented a completely new page table representation there,
> which should solve this problem once and for all.

Backporting all that hash table code into OCaml 3.10 didn't look like
fun to me :-(

Rich.

-- 
Richard Jones
Red Hat