memory usage

memory usage

[Jean Krivine]

Dear list members,

I am trying to run a stochastic simulator (written in ocaml) on a huge
data set and I have the following error message:

sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Fatal error: out of memory.

My system:

Mac Pro running OS X 10.5.4
Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
Memory:  10 GB 800 MHz DDR2 FB-DIMM

Does someone know what happened? Do you have any idea of any parameter
I could tune in order to avoid that?

Thank you very much!

Jean

Re: [Caml-list] memory usage

[Till Varoquaux]

It is hard to tell without any more informations but sometimes the
garbage collector needs some gentle proding:

OCaml handles it's own memory but can be a bad citizen when it comes
to making room for others. Unfortunately ocaml also has a bit of a
double personality: it doesn't know much about resources used in
external libraries or even in some of its own library (e.g. on a 32
bits machine running out of addressable space because of
Bigarray.map_file is not unheard of).

If this is your problem, you can either sprinkle your source code with
calls to Gc.major or tweak it using Gc.set.

Till

On Fri, Jul 11, 2008 at 8:49 PM, Jean Krivine
<jean_krivine@hms.harvard.edu> wrote:
> Dear list members,
>
> I am trying to run a stochastic simulator (written in ocaml) on a huge
> data set and I have the following error message:
>
> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Fatal error: out of memory.
>
> My system:
>
> Mac Pro running OS X 10.5.4
> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>
> Does someone know what happened? Do you have any idea of any parameter
> I could tune in order to avoid that?
>
> Thank you very much!
>
> Jean
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>



-- 
http://till-varoquaux.blogspot.com/

Re: [Caml-list] memory usage

[Richard Jones]

On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
> I am trying to run a stochastic simulator (written in ocaml) on a huge
> data set and I have the following error message:

I can confirm that OCaml works fine with huge datasets, on 64 bit
platforms anyway.

> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
> *** error: can't allocate region
> *** set a breakpoint in malloc_error_break to debug
> Fatal error: out of memory.
> 
> My system:
> 
> Mac Pro running OS X 10.5.4
> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
> Memory:  10 GB 800 MHz DDR2 FB-DIMM
> 
> Does someone know what happened? Do you have any idea of any parameter
> I could tune in order to avoid that?

Is the compiler 32 bits or 64 bits on this machine?  Try doing:

  $ ocaml
  # Sys.word_size ;;

It should print out either '32' or '64'.

Also run your program under whatever the OS X equivalent of 'strace'
is (ktrace?) to find out exactly why the mmap call fails.

OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
and randomized address spaces
(https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it doesn't
seem like this is the same issue.

Rich.

-- 
Richard Jones
Red Hat

Re: [Caml-list] memory usage

[Jean Krivine]

Dear all

I downloaded the last version of ocaml (3.10.2) but I must confess I
don't know what option I should pass to the compiler to make a binary
that uses 64 bits.
I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
work. Any idea?



On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org> wrote:
> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>> I am trying to run a stochastic simulator (written in ocaml) on a huge
>> data set and I have the following error message:
>
> I can confirm that OCaml works fine with huge datasets, on 64 bit
> platforms anyway.
>
>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>> *** error: can't allocate region
>> *** set a breakpoint in malloc_error_break to debug
>> Fatal error: out of memory.
>>
>> My system:
>>
>> Mac Pro running OS X 10.5.4
>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>
>> Does someone know what happened? Do you have any idea of any parameter
>> I could tune in order to avoid that?
>
> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>
>  $ ocaml
>  # Sys.word_size ;;
>
> It should print out either '32' or '64'.
>
> Also run your program under whatever the OS X equivalent of 'strace'
> is (ktrace?) to find out exactly why the mmap call fails.
>
> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
> and randomized address spaces
> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it doesn't
> seem like this is the same issue.
>
> Rich.
>
> --
> Richard Jones
> Red Hat
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs
>

Re: [Caml-list] memory usage

[Andres Varon]

Hello Jean,

There is no 64-bit native OCaml compiler for Mac OS X intel.  I have a  
patch that works in Leopard, but did not compile opt.opt in Tiger,  
meaning that something is not OK,  so I did not offer it to the  
community. The bootstrap went fine, findlib and godi compiled OK too.  
I can post the patches somewhere if you want to give it a shot.

My memory intensive application runs fine in Leopard with this  
compiler. But the binaries do not execute in Tiger (I found that other  
people had the same trouble copying a 64 bit apps from Leopard to  
Tiger and the other way around, but didn't look into it).

If you want it ... I can post it, maybe someone can cleanup my job?  
All that would be needed after patching is:

./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental

(The prefix I always add for my ocaml-modified comilers).

best,

Andres

On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:

> Dear all
>
> I downloaded the last version of ocaml (3.10.2) but I must confess I
> don't know what option I should pass to the compiler to make a binary
> that uses 64 bits.
> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
> work. Any idea?
>
>
>
> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org>  
> wrote:
>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>> I am trying to run a stochastic simulator (written in ocaml) on a  
>>> huge
>>> data set and I have the following error message:
>>
>> I can confirm that OCaml works fine with huge datasets, on 64 bit
>> platforms anyway.
>>
>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>> *** error: can't allocate region
>>> *** set a breakpoint in malloc_error_break to debug
>>> Fatal error: out of memory.
>>>
>>> My system:
>>>
>>> Mac Pro running OS X 10.5.4
>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>
>>> Does someone know what happened? Do you have any idea of any  
>>> parameter
>>> I could tune in order to avoid that?
>>
>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>
>> $ ocaml
>> # Sys.word_size ;;
>>
>> It should print out either '32' or '64'.
>>
>> Also run your program under whatever the OS X equivalent of 'strace'
>> is (ktrace?) to find out exactly why the mmap call fails.
>>
>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
>> and randomized address spaces
>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it  
>> doesn't
>> seem like this is the same issue.
>>
>> Rich.
>>
>> --
>> Richard Jones
>> Red Hat
>>
>> _______________________________________________
>> Caml-list mailing list. Subscription management:
>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>> Archives: http://caml.inria.fr
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>
> _______________________________________________
> Caml-list mailing list. Subscription management:
> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
> Archives: http://caml.inria.fr
> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
> Bug reports: http://caml.inria.fr/bin/caml-bugs

Re: [Caml-list] memory usage

[Jean Krivine]

I'd be glad to try the patch if you could post it somewhere!

J

On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <avaron@gmail.com> wrote:
> Hello Jean,
>
> There is no 64-bit native OCaml compiler for Mac OS X intel.  I have a patch
> that works in Leopard, but did not compile opt.opt in Tiger, meaning that
> something is not OK,  so I did not offer it to the community. The bootstrap
> went fine, findlib and godi compiled OK too. I can post the patches
> somewhere if you want to give it a shot.
>
> My memory intensive application runs fine in Leopard with this compiler. But
> the binaries do not execute in Tiger (I found that other people had the same
> trouble copying a 64 bit apps from Leopard to Tiger and the other way
> around, but didn't look into it).
>
> If you want it ... I can post it, maybe someone can cleanup my job? All that
> would be needed after patching is:
>
> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental
>
> (The prefix I always add for my ocaml-modified comilers).
>
> best,
>
> Andres
>
> On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:
>
>> Dear all
>>
>> I downloaded the last version of ocaml (3.10.2) but I must confess I
>> don't know what option I should pass to the compiler to make a binary
>> that uses 64 bits.
>> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
>> work. Any idea?
>>
>>
>>
>> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org> wrote:
>>>
>>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>>>
>>>> I am trying to run a stochastic simulator (written in ocaml) on a huge
>>>> data set and I have the following error message:
>>>
>>> I can confirm that OCaml works fine with huge datasets, on 64 bit
>>> platforms anyway.
>>>
>>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>>> *** error: can't allocate region
>>>> *** set a breakpoint in malloc_error_break to debug
>>>> Fatal error: out of memory.
>>>>
>>>> My system:
>>>>
>>>> Mac Pro running OS X 10.5.4
>>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>>
>>>> Does someone know what happened? Do you have any idea of any parameter
>>>> I could tune in order to avoid that?
>>>
>>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>>
>>> $ ocaml
>>> # Sys.word_size ;;
>>>
>>> It should print out either '32' or '64'.
>>>
>>> Also run your program under whatever the OS X equivalent of 'strace'
>>> is (ktrace?) to find out exactly why the mmap call fails.
>>>
>>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
>>> and randomized address spaces
>>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it doesn't
>>> seem like this is the same issue.
>>>
>>> Rich.
>>>
>>> --
>>> Richard Jones
>>> Red Hat
>>>
>>> _______________________________________________
>>> Caml-list mailing list. Subscription management:
>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>> Archives: http://caml.inria.fr
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>
>> _______________________________________________
>> Caml-list mailing list. Subscription management:
>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>> Archives: http://caml.inria.fr
>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>
>

Re: [Caml-list] memory usage

[Andres Varon]

On Jul 15, 2008, at 3:38 PM, Jean Krivine wrote:

> I'd be glad to try the patch if you could post it somewhere!

I have posted it in:

http://research.amnh.org/~avaron/ocaml/

best,

Andres
>
> J
>
> On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <avaron@gmail.com>  
> wrote:
>> Hello Jean,
>>
>> There is no 64-bit native OCaml compiler for Mac OS X intel.  I  
>> have a patch
>> that works in Leopard, but did not compile opt.opt in Tiger,  
>> meaning that
>> something is not OK,  so I did not offer it to the community. The  
>> bootstrap
>> went fine, findlib and godi compiled OK too. I can post the patches
>> somewhere if you want to give it a shot.
>>
>> My memory intensive application runs fine in Leopard with this  
>> compiler. But
>> the binaries do not execute in Tiger (I found that other people had  
>> the same
>> trouble copying a 64 bit apps from Leopard to Tiger and the other way
>> around, but didn't look into it).
>>
>> If you want it ... I can post it, maybe someone can cleanup my job?  
>> All that
>> would be needed after patching is:
>>
>> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental
>>
>> (The prefix I always add for my ocaml-modified comilers).
>>
>> best,
>>
>> Andres
>>
>> On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:
>>
>>> Dear all
>>>
>>> I downloaded the last version of ocaml (3.10.2) but I must confess I
>>> don't know what option I should pass to the compiler to make a  
>>> binary
>>> that uses 64 bits.
>>> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
>>> work. Any idea?
>>>
>>>
>>>
>>> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org>  
>>> wrote:
>>>>
>>>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>>>>
>>>>> I am trying to run a stochastic simulator (written in ocaml) on  
>>>>> a huge
>>>>> data set and I have the following error message:
>>>>
>>>> I can confirm that OCaml works fine with huge datasets, on 64 bit
>>>> platforms anyway.
>>>>
>>>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>>>> *** error: can't allocate region
>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>> Fatal error: out of memory.
>>>>>
>>>>> My system:
>>>>>
>>>>> Mac Pro running OS X 10.5.4
>>>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>>>
>>>>> Does someone know what happened? Do you have any idea of any  
>>>>> parameter
>>>>> I could tune in order to avoid that?
>>>>
>>>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>>>
>>>> $ ocaml
>>>> # Sys.word_size ;;
>>>>
>>>> It should print out either '32' or '64'.
>>>>
>>>> Also run your program under whatever the OS X equivalent of  
>>>> 'strace'
>>>> is (ktrace?) to find out exactly why the mmap call fails.
>>>>
>>>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of  
>>>> mmap
>>>> and randomized address spaces
>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it  
>>>> doesn't
>>>> seem like this is the same issue.
>>>>
>>>> Rich.
>>>>
>>>> --
>>>> Richard Jones
>>>> Red Hat
>>>>
>>>> _______________________________________________
>>>> Caml-list mailing list. Subscription management:
>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>> Archives: http://caml.inria.fr
>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>
>>>
>>> _______________________________________________
>>> Caml-list mailing list. Subscription management:
>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>> Archives: http://caml.inria.fr
>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>
>>

Re: [Caml-list] memory usage

[Jean Krivine]

Great thanks!

J

On Wed, Jul 16, 2008 at 10:16 AM, Andres Varon <avaron@gmail.com> wrote:
>
> On Jul 15, 2008, at 3:38 PM, Jean Krivine wrote:
>
>> I'd be glad to try the patch if you could post it somewhere!
>
> I have posted it in:
>
> http://research.amnh.org/~avaron/ocaml/
>
> best,
>
> Andres
>>
>> J
>>
>> On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <avaron@gmail.com> wrote:
>>>
>>> Hello Jean,
>>>
>>> There is no 64-bit native OCaml compiler for Mac OS X intel.  I have a
>>> patch
>>> that works in Leopard, but did not compile opt.opt in Tiger, meaning that
>>> something is not OK,  so I did not offer it to the community. The
>>> bootstrap
>>> went fine, findlib and godi compiled OK too. I can post the patches
>>> somewhere if you want to give it a shot.
>>>
>>> My memory intensive application runs fine in Leopard with this compiler.
>>> But
>>> the binaries do not execute in Tiger (I found that other people had the
>>> same
>>> trouble copying a 64 bit apps from Leopard to Tiger and the other way
>>> around, but didn't look into it).
>>>
>>> If you want it ... I can post it, maybe someone can cleanup my job? All
>>> that
>>> would be needed after patching is:
>>>
>>> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental
>>>
>>> (The prefix I always add for my ocaml-modified comilers).
>>>
>>> best,
>>>
>>> Andres
>>>
>>> On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:
>>>
>>>> Dear all
>>>>
>>>> I downloaded the last version of ocaml (3.10.2) but I must confess I
>>>> don't know what option I should pass to the compiler to make a binary
>>>> that uses 64 bits.
>>>> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
>>>> work. Any idea?
>>>>
>>>>
>>>>
>>>> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org> wrote:
>>>>>
>>>>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>>>>>
>>>>>> I am trying to run a stochastic simulator (written in ocaml) on a huge
>>>>>> data set and I have the following error message:
>>>>>
>>>>> I can confirm that OCaml works fine with huge datasets, on 64 bit
>>>>> platforms anyway.
>>>>>
>>>>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>>>>> *** error: can't allocate region
>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>> Fatal error: out of memory.
>>>>>>
>>>>>> My system:
>>>>>>
>>>>>> Mac Pro running OS X 10.5.4
>>>>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>>>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>>>>
>>>>>> Does someone know what happened? Do you have any idea of any parameter
>>>>>> I could tune in order to avoid that?
>>>>>
>>>>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>>>>
>>>>> $ ocaml
>>>>> # Sys.word_size ;;
>>>>>
>>>>> It should print out either '32' or '64'.
>>>>>
>>>>> Also run your program under whatever the OS X equivalent of 'strace'
>>>>> is (ktrace?) to find out exactly why the mmap call fails.
>>>>>
>>>>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
>>>>> and randomized address spaces
>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it doesn't
>>>>> seem like this is the same issue.
>>>>>
>>>>> Rich.
>>>>>
>>>>> --
>>>>> Richard Jones
>>>>> Red Hat
>>>>>
>>>>> _______________________________________________
>>>>> Caml-list mailing list. Subscription management:
>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>> Archives: http://caml.inria.fr
>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>>
>>>>
>>>> _______________________________________________
>>>> Caml-list mailing list. Subscription management:
>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>> Archives: http://caml.inria.fr
>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>
>>>
>
>

Re: [Caml-list] memory usage

[Jean Krivine]

Good news, I just tested the patch and it works great with my application!
I just had to modify the module random since a call to (Random.int
max_int) may raise and exception (it is made for 32 bits integers).
So I guess that modification should be included in the patch.

Thanks a lot Andres.
Jean

On Wed, Jul 16, 2008 at 12:27 PM, Jean Krivine
<jean_krivine@hms.harvard.edu> wrote:
> Great thanks!
>
> J
>
> On Wed, Jul 16, 2008 at 10:16 AM, Andres Varon <avaron@gmail.com> wrote:
>>
>> On Jul 15, 2008, at 3:38 PM, Jean Krivine wrote:
>>
>>> I'd be glad to try the patch if you could post it somewhere!
>>
>> I have posted it in:
>>
>> http://research.amnh.org/~avaron/ocaml/
>>
>> best,
>>
>> Andres
>>>
>>> J
>>>
>>> On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <avaron@gmail.com> wrote:
>>>>
>>>> Hello Jean,
>>>>
>>>> There is no 64-bit native OCaml compiler for Mac OS X intel.  I have a
>>>> patch
>>>> that works in Leopard, but did not compile opt.opt in Tiger, meaning that
>>>> something is not OK,  so I did not offer it to the community. The
>>>> bootstrap
>>>> went fine, findlib and godi compiled OK too. I can post the patches
>>>> somewhere if you want to give it a shot.
>>>>
>>>> My memory intensive application runs fine in Leopard with this compiler.
>>>> But
>>>> the binaries do not execute in Tiger (I found that other people had the
>>>> same
>>>> trouble copying a 64 bit apps from Leopard to Tiger and the other way
>>>> around, but didn't look into it).
>>>>
>>>> If you want it ... I can post it, maybe someone can cleanup my job? All
>>>> that
>>>> would be needed after patching is:
>>>>
>>>> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental
>>>>
>>>> (The prefix I always add for my ocaml-modified comilers).
>>>>
>>>> best,
>>>>
>>>> Andres
>>>>
>>>> On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:
>>>>
>>>>> Dear all
>>>>>
>>>>> I downloaded the last version of ocaml (3.10.2) but I must confess I
>>>>> don't know what option I should pass to the compiler to make a binary
>>>>> that uses 64 bits.
>>>>> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
>>>>> work. Any idea?
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org> wrote:
>>>>>>
>>>>>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>>>>>>
>>>>>>> I am trying to run a stochastic simulator (written in ocaml) on a huge
>>>>>>> data set and I have the following error message:
>>>>>>
>>>>>> I can confirm that OCaml works fine with huge datasets, on 64 bit
>>>>>> platforms anyway.
>>>>>>
>>>>>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>>>>>> *** error: can't allocate region
>>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>>> Fatal error: out of memory.
>>>>>>>
>>>>>>> My system:
>>>>>>>
>>>>>>> Mac Pro running OS X 10.5.4
>>>>>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>>>>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>>>>>
>>>>>>> Does someone know what happened? Do you have any idea of any parameter
>>>>>>> I could tune in order to avoid that?
>>>>>>
>>>>>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>>>>>
>>>>>> $ ocaml
>>>>>> # Sys.word_size ;;
>>>>>>
>>>>>> It should print out either '32' or '64'.
>>>>>>
>>>>>> Also run your program under whatever the OS X equivalent of 'strace'
>>>>>> is (ktrace?) to find out exactly why the mmap call fails.
>>>>>>
>>>>>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of mmap
>>>>>> and randomized address spaces
>>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it doesn't
>>>>>> seem like this is the same issue.
>>>>>>
>>>>>> Rich.
>>>>>>
>>>>>> --
>>>>>> Richard Jones
>>>>>> Red Hat
>>>>>>
>>>>>> _______________________________________________
>>>>>> Caml-list mailing list. Subscription management:
>>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>>> Archives: http://caml.inria.fr
>>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Caml-list mailing list. Subscription management:
>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>> Archives: http://caml.inria.fr
>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>
>>>>
>>
>>
>

Re: [Caml-list] memory usage

[Andres Varon]

On Jul 16, 2008, at 2:07 PM, Jean Krivine wrote:

> Good news, I just tested the patch and it works great with my  
> application!
> I just had to modify the module random since a call to (Random.int
> max_int) may raise and exception (it is made for 32 bits integers).
> So I guess that modification should be included in the patch.

I don't think that's a good idea. You have to use Random.int64 to get  
a 64 bit random integer. The Random.int function will return an  
integer between 0 and 2^30. Check the Random module documentation here:

http://caml.inria.fr/pub/docs/manual-ocaml/libref/Random.html

I wouldn't play with a random number generator unless I know exactly  
what I'm doing. Your results depend on it! (well, your messed-up-by- 
andres compiler could already have issues ... :-(, for what I use it I  
can verify the result with a 32 bit binary or a 64 bit linux binary,  
if you can, then do the same!).


Andres
>
>
> Thanks a lot Andres.
> Jean
>
> On Wed, Jul 16, 2008 at 12:27 PM, Jean Krivine
> <jean_krivine@hms.harvard.edu> wrote:
>> Great thanks!
>>
>> J
>>
>> On Wed, Jul 16, 2008 at 10:16 AM, Andres Varon <avaron@gmail.com>  
>> wrote:
>>>
>>> On Jul 15, 2008, at 3:38 PM, Jean Krivine wrote:
>>>
>>>> I'd be glad to try the patch if you could post it somewhere!
>>>
>>> I have posted it in:
>>>
>>> http://research.amnh.org/~avaron/ocaml/
>>>
>>> best,
>>>
>>> Andres
>>>>
>>>> J
>>>>
>>>> On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <avaron@gmail.com>  
>>>> wrote:
>>>>>
>>>>> Hello Jean,
>>>>>
>>>>> There is no 64-bit native OCaml compiler for Mac OS X intel.  I  
>>>>> have a
>>>>> patch
>>>>> that works in Leopard, but did not compile opt.opt in Tiger,  
>>>>> meaning that
>>>>> something is not OK,  so I did not offer it to the community. The
>>>>> bootstrap
>>>>> went fine, findlib and godi compiled OK too. I can post the  
>>>>> patches
>>>>> somewhere if you want to give it a shot.
>>>>>
>>>>> My memory intensive application runs fine in Leopard with this  
>>>>> compiler.
>>>>> But
>>>>> the binaries do not execute in Tiger (I found that other people  
>>>>> had the
>>>>> same
>>>>> trouble copying a 64 bit apps from Leopard to Tiger and the  
>>>>> other way
>>>>> around, but didn't look into it).
>>>>>
>>>>> If you want it ... I can post it, maybe someone can cleanup my  
>>>>> job? All
>>>>> that
>>>>> would be needed after patching is:
>>>>>
>>>>> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/ 
>>>>> experimental
>>>>>
>>>>> (The prefix I always add for my ocaml-modified comilers).
>>>>>
>>>>> best,
>>>>>
>>>>> Andres
>>>>>
>>>>> On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:
>>>>>
>>>>>> Dear all
>>>>>>
>>>>>> I downloaded the last version of ocaml (3.10.2) but I must  
>>>>>> confess I
>>>>>> don't know what option I should pass to the compiler to make a  
>>>>>> binary
>>>>>> that uses 64 bits.
>>>>>> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that  
>>>>>> doesn't
>>>>>> work. Any idea?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones  
>>>>>> <rich@annexia.org> wrote:
>>>>>>>
>>>>>>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>>>>>>>
>>>>>>>> I am trying to run a stochastic simulator (written in ocaml)  
>>>>>>>> on a huge
>>>>>>>> data set and I have the following error message:
>>>>>>>
>>>>>>> I can confirm that OCaml works fine with huge datasets, on 64  
>>>>>>> bit
>>>>>>> platforms anyway.
>>>>>>>
>>>>>>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>>>>>>> *** error: can't allocate region
>>>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>>>> Fatal error: out of memory.
>>>>>>>>
>>>>>>>> My system:
>>>>>>>>
>>>>>>>> Mac Pro running OS X 10.5.4
>>>>>>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>>>>>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>>>>>>
>>>>>>>> Does someone know what happened? Do you have any idea of any  
>>>>>>>> parameter
>>>>>>>> I could tune in order to avoid that?
>>>>>>>
>>>>>>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>>>>>>
>>>>>>> $ ocaml
>>>>>>> # Sys.word_size ;;
>>>>>>>
>>>>>>> It should print out either '32' or '64'.
>>>>>>>
>>>>>>> Also run your program under whatever the OS X equivalent of  
>>>>>>> 'strace'
>>>>>>> is (ktrace?) to find out exactly why the mmap call fails.
>>>>>>>
>>>>>>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use  
>>>>>>> of mmap
>>>>>>> and randomized address spaces
>>>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it  
>>>>>>> doesn't
>>>>>>> seem like this is the same issue.
>>>>>>>
>>>>>>> Rich.
>>>>>>>
>>>>>>> --
>>>>>>> Richard Jones
>>>>>>> Red Hat
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Caml-list mailing list. Subscription management:
>>>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>>>> Archives: http://caml.inria.fr
>>>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Caml-list mailing list. Subscription management:
>>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>>> Archives: http://caml.inria.fr
>>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>>
>>>>>
>>>
>>>
>>

Re: [Caml-list] memory usage

[Jean Krivine]

I agree. I should use Int64 instead of just int, but I still think
that the application (Random.int max_int) should not be exception
prone. Since max_int is architecture dependent, then so should be
Random.int no?

But you point is well taken. Thanks again
J


On Wed, Jul 16, 2008 at 2:44 PM, Andres Varon <avaron@gmail.com> wrote:
>
>
> On Jul 16, 2008, at 2:07 PM, Jean Krivine wrote:
>
>> Good news, I just tested the patch and it works great with my application!
>> I just had to modify the module random since a call to (Random.int
>> max_int) may raise and exception (it is made for 32 bits integers).
>> So I guess that modification should be included in the patch.
>
> I don't think that's a good idea. You have to use Random.int64 to get a 64
> bit random integer. The Random.int function will return an integer between 0
> and 2^30. Check the Random module documentation here:
>
> http://caml.inria.fr/pub/docs/manual-ocaml/libref/Random.html
>
> I wouldn't play with a random number generator unless I know exactly what
> I'm doing. Your results depend on it! (well, your messed-up-by-andres
> compiler could already have issues ... :-(, for what I use it I can verify
> the result with a 32 bit binary or a 64 bit linux binary, if you can, then
> do the same!).
>
>
> Andres
>>
>>
>> Thanks a lot Andres.
>> Jean
>>
>> On Wed, Jul 16, 2008 at 12:27 PM, Jean Krivine
>> <jean_krivine@hms.harvard.edu> wrote:
>>>
>>> Great thanks!
>>>
>>> J
>>>
>>> On Wed, Jul 16, 2008 at 10:16 AM, Andres Varon <avaron@gmail.com> wrote:
>>>>
>>>> On Jul 15, 2008, at 3:38 PM, Jean Krivine wrote:
>>>>
>>>>> I'd be glad to try the patch if you could post it somewhere!
>>>>
>>>> I have posted it in:
>>>>
>>>> http://research.amnh.org/~avaron/ocaml/
>>>>
>>>> best,
>>>>
>>>> Andres
>>>>>
>>>>> J
>>>>>
>>>>> On Tue, Jul 15, 2008 at 3:31 PM, Andres Varon <avaron@gmail.com> wrote:
>>>>>>
>>>>>> Hello Jean,
>>>>>>
>>>>>> There is no 64-bit native OCaml compiler for Mac OS X intel.  I have a
>>>>>> patch
>>>>>> that works in Leopard, but did not compile opt.opt in Tiger, meaning
>>>>>> that
>>>>>> something is not OK,  so I did not offer it to the community. The
>>>>>> bootstrap
>>>>>> went fine, findlib and godi compiled OK too. I can post the patches
>>>>>> somewhere if you want to give it a shot.
>>>>>>
>>>>>> My memory intensive application runs fine in Leopard with this
>>>>>> compiler.
>>>>>> But
>>>>>> the binaries do not execute in Tiger (I found that other people had
>>>>>> the
>>>>>> same
>>>>>> trouble copying a 64 bit apps from Leopard to Tiger and the other way
>>>>>> around, but didn't look into it).
>>>>>>
>>>>>> If you want it ... I can post it, maybe someone can cleanup my job?
>>>>>> All
>>>>>> that
>>>>>> would be needed after patching is:
>>>>>>
>>>>>> ./configure -host x86_64-apple-darwin -prefix /opt/ocaml/experimental
>>>>>>
>>>>>> (The prefix I always add for my ocaml-modified comilers).
>>>>>>
>>>>>> best,
>>>>>>
>>>>>> Andres
>>>>>>
>>>>>> On Jul 15, 2008, at 1:06 PM, Jean Krivine wrote:
>>>>>>
>>>>>>> Dear all
>>>>>>>
>>>>>>> I downloaded the last version of ocaml (3.10.2) but I must confess I
>>>>>>> don't know what option I should pass to the compiler to make a binary
>>>>>>> that uses 64 bits.
>>>>>>> I tried naively ocamlopt -ccopt -arch -ccopt x86_64 but that doesn't
>>>>>>> work. Any idea?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 11, 2008 at 6:01 PM, Richard Jones <rich@annexia.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Fri, Jul 11, 2008 at 03:49:26PM -0400, Jean Krivine wrote:
>>>>>>>>>
>>>>>>>>> I am trying to run a stochastic simulator (written in ocaml) on a
>>>>>>>>> huge
>>>>>>>>> data set and I have the following error message:
>>>>>>>>
>>>>>>>> I can confirm that OCaml works fine with huge datasets, on 64 bit
>>>>>>>> platforms anyway.
>>>>>>>>
>>>>>>>>> sim(9595) malloc: *** mmap(size=1048576) failed (error code=12)
>>>>>>>>> *** error: can't allocate region
>>>>>>>>> *** set a breakpoint in malloc_error_break to debug
>>>>>>>>> Fatal error: out of memory.
>>>>>>>>>
>>>>>>>>> My system:
>>>>>>>>>
>>>>>>>>> Mac Pro running OS X 10.5.4
>>>>>>>>> Processor:  2 x 2.8 GHz Quad-Core Intel Xeon
>>>>>>>>> Memory:  10 GB 800 MHz DDR2 FB-DIMM
>>>>>>>>>
>>>>>>>>> Does someone know what happened? Do you have any idea of any
>>>>>>>>> parameter
>>>>>>>>> I could tune in order to avoid that?
>>>>>>>>
>>>>>>>> Is the compiler 32 bits or 64 bits on this machine?  Try doing:
>>>>>>>>
>>>>>>>> $ ocaml
>>>>>>>> # Sys.word_size ;;
>>>>>>>>
>>>>>>>> It should print out either '32' or '64'.
>>>>>>>>
>>>>>>>> Also run your program under whatever the OS X equivalent of 'strace'
>>>>>>>> is (ktrace?) to find out exactly why the mmap call fails.
>>>>>>>>
>>>>>>>> OCaml <= 3.10.2 on Linux suffers a nasty problem with its use of
>>>>>>>> mmap
>>>>>>>> and randomized address spaces
>>>>>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=445545#c9) but it
>>>>>>>> doesn't
>>>>>>>> seem like this is the same issue.
>>>>>>>>
>>>>>>>> Rich.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Richard Jones
>>>>>>>> Red Hat
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Caml-list mailing list. Subscription management:
>>>>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>>>>> Archives: http://caml.inria.fr
>>>>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Caml-list mailing list. Subscription management:
>>>>>>> http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
>>>>>>> Archives: http://caml.inria.fr
>>>>>>> Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
>>>>>>> Bug reports: http://caml.inria.fr/bin/caml-bugs
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>
>

Re: memory usage

[Sylvain Le Gall]

Can you make sure that all your function terminate and are joined
(Thread.join). I think this will help to make sure that the thread exit
and call thread_kill (see OCaml source code).

If you take a look at thread_kill there is a function stat_free + set to
NULL things called stack_low, stack_high... Maybe all the data you are
seeing come from this...

I am not sure that Thread.join will free anything, but it will help you
to be sure that your thread has exited correctly.

Regards,
Sylvain Le Gall

Re: memory usage

[Sylvain Le Gall]

On 12-01-2009, John Lepikhin <john@ispsystem.com> wrote:
>
>> > Each thread is killed after work is done
>> How do you "kill" the threads? I hope this is just a figure of
>> speech for "I do an orderly shutdown of each thread after work is done."
>
> Well, that was consequence of my bad English :-) Threads finish their
> work and exit. I also made a simple wrapper to Thread.create to be sure
> that all work inside threads is done:
>
> module MyThread =
>   let create f p =
>     let dowork _ =
>       (* log thread creation *)
>       f p;
>       (* log thread shutdown *)
>     in
>     Thread.create dowork ()
> end

Do you use some kind of Thread.join ? 

Regards
Sylvain Le Gall

Re: memory usage

[Sylvain Le Gall]

On 12-01-2009, John Lepikhin <john@ispsystem.com> wrote:
>> Starting point should be to call this periodically:
>> 
>>       Gc.compact ();
>>       let stat = Gc.stat () in
>>       let live_words = stat.Gc.live_words in
>>       eprintf "live words %d\n%!" live_words;
>> 
>> which will tell you how many words (ie 4 or 8 byte chunks) are
>> reachable according to the garbage collector.
>
> Richard, here is result (statistics was saved every 10 seconds):
>
> live words - RSS:
>
> 186980 - 12380KB <-- after first 10 seconds of work
> 154156 - 18232KB
> 153923 - 19648KB
> ...
> after 10 minutes of work:
> 203842 - 33436KB
> 170559 - 33528KB
> 187018 - 33664KB
> 71626 - 33592KB
>
> Sometimes live words drops down to 40.000. But RSS always stay near
> 30-50MB.
>

To get real memory used, (Sys.word_size * live_word / 8). Do you use
out-of-heap datastructure that can use memory ? (malloc-ed
datastructure).

Regards,
Sylvain Le Gall

memory usage

[John Lepikhin]

Hello,

I have developed an application and have issue with memory usage.
Application was designed for work as server, 24x7. After several hours
of work, RSS memory grows up to 40-50MB. Each thread must not use more
than 100-200KB of memory (maximum 20 threads are run at the same time).
GC debug information:

Growing heap to 3840k bytes
Growing heap to 4320k bytes
Shrinking heap to 3840k bytes
Growing heap to 4320k bytes
Growing heap to 4800k bytes
Growing heap to 5280k bytes
Shrinking heap to 4800k bytes
Shrinking heap to 4320k bytes
Shrinking heap to 3840k bytes
Growing heap to 4320k bytes
Growing heap to 4800k bytes
Growing heap to 5280k bytes
Shrinking heap to 4800k bytes
Shrinking heap to 4320k bytes
Shrinking heap to 3840k bytes
Shrinking heap to 3360k bytes
Growing heap to 3840k bytes
Growing heap to 4320k bytes
Growing heap to 4800k bytes
Growing heap to 960k bytes
Shrinking heap to 4320k bytes
Shrinking heap to 3840k bytes
Shrinking heap to 3360k bytes
Shrinking heap to 2880k bytes
Growing heap to 3360k bytes
Shrinking heap to 2880k bytes
Shrinking heap to 2400k bytes
Growing heap to 2880k bytes
Growing heap to 3360k bytes
Growing heap to 3840k bytes
Shrinking heap to 3360k bytes
Shrinking heap to 2880k bytes
Growing heap to 3360k bytes
Shrinking heap to 2880k bytes

As you can see, heap size never gets bigger 5-6MB. I made core dump of
process. 90% of memory was filled with values which are created inside
threads and never been copied outside of them. Each thread is killed
after work is done, I am absolutely sure in it; all unused file
descriptors are closed. Playing with Gc.set has not brought notable
results.

Please, give me some start point to find the roots of the problem.

Sorry for my English.

Re: Memory usage

[David Mentré]

"Dorlet Emmanuel" <Emmanuel.Dorlet@cea.fr> writes:

> Does any one has a good and simple suggestion?

To monitor global memory usage, I use the following code :

# let stat = Gc.stat ()
  and control = Gc.get () in
  let max_words_total = stat.Gc.heap_words + control.Gc.minor_heap_size in
  Printf.printf "Maximum memory used (allocated): %d kBytes\n" 
          (max_words_total * Sys.word_size / 8 / 1024);;

  Maximum memory used (allocated): 624 kBytes
- : unit = ()

It gives you the total heap allocated.

Moreover, at the following URL, you'll find a program, "size", to
compute memory size of an OCaml value :

 http://www.lri.fr/~filliatr/software.en.html


If you have more precise ways to know memory behavior, I would be very
interested in them.

Best regards,
d.
-- 
 David.Mentre@irisa.fr -- http://www.irisa.fr/prive/dmentre/
 Opinions expressed here are only mine.

Memory usage

[Dorlet Emmanuel]

Hello,

I am in charge of a very very simple test on the ability of Ocaml
garbage to free unused memory.
Obvioulsy, Ocaml does the work properly (but not in the case of the
global context, where every variable allocation is consider as a
living reference, even if that is not true).
But I can't see, from my (perhaps too fast) reading of the documentation,
how to compute a global memory usage and how I can monitor it.
Does any one has a good and simple suggestion?


__________________________________________________________

  Emmanuel DORLET
  Commissariat à l'Energie Atomique
  Centre de Saclay - DRN-DMT/SYSCO/LGLS
  Bâtiment 460 - Pièce 19 A
  Téléphone : 01 69 08 91 17
  Télécopie : 01 69 08 96 96
  e-mail: Emmanuel.Dorlet@cea.fr
__________________________________________________________