* RE: [Caml-list] Ocaml crash in windows due to running out of memory
@ 2013-11-20 10:52 Amogh Margoor
2013-11-20 20:56 ` Adrien Nader
0 siblings, 1 reply; 10+ messages in thread
From: Amogh Margoor @ 2013-11-20 10:52 UTC (permalink / raw)
To: caml-list
[-- Attachment #1: Type: text/plain, Size: 427 bytes --]
Hi,
The Ocaml version I am facing problem with is 3.12.1.
Regards,
Amogh
From: Amogh Margoor
Sent: Wednesday, November 20, 2013 4:19 PM
To: 'caml-list@inria.fr'
Subject: Ocaml crash in windows due to running out of memory
Hi,
We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
Is that a known issue ?
Regards,
Amogh
[-- Attachment #2: Type: text/html, Size: 3365 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-20 10:52 [Caml-list] Ocaml crash in windows due to running out of memory Amogh Margoor
@ 2013-11-20 20:56 ` Adrien Nader
2013-11-25 13:01 ` Goswin von Brederlow
0 siblings, 1 reply; 10+ messages in thread
From: Adrien Nader @ 2013-11-20 20:56 UTC (permalink / raw)
To: Amogh Margoor; +Cc: caml-list
Hi,
On Wed, Nov 20, 2013, Amogh Margoor wrote:
> Hi,
> The Ocaml version I am facing problem with is 3.12.1.
> Regards,
> Amogh
>
> From: Amogh Margoor
> Sent: Wednesday, November 20, 2013 4:19 PM
> To: 'caml-list@inria.fr'
> Subject: Ocaml crash in windows due to running out of memory
>
> Hi,
> We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
> Is that a known issue ?
What is the exception and how much memory is your application using at
that moment?
--
Adrien Nader
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-20 20:56 ` Adrien Nader
@ 2013-11-25 13:01 ` Goswin von Brederlow
2013-11-25 13:55 ` Adrien Nader
0 siblings, 1 reply; 10+ messages in thread
From: Goswin von Brederlow @ 2013-11-25 13:01 UTC (permalink / raw)
To: Adrien Nader; +Cc: Amogh Margoor, caml-list
On Wed, Nov 20, 2013 at 09:56:11PM +0100, Adrien Nader wrote:
> Hi,
>
> On Wed, Nov 20, 2013, Amogh Margoor wrote:
> > Hi,
> > The Ocaml version I am facing problem with is 3.12.1.
> > Regards,
> > Amogh
> >
> > From: Amogh Margoor
> > Sent: Wednesday, November 20, 2013 4:19 PM
> > To: 'caml-list@inria.fr'
> > Subject: Ocaml crash in windows due to running out of memory
> >
> > Hi,
> > We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
> > Is that a known issue ?
>
> What is the exception and how much memory is your application using at
> that moment?
Many kernels only allocate address space on malloc() and assign pages
only on use. They also support over commit of memory. Meaning
applications can allocate more memory than the system has under the
assumption that they won't actually use it all (or that maybe some
memory will be freed before the apps actually use the memory).
Due to that in many kernels malloc() basically never fails but when
you use the memory and the kernel can't find any page to assign to it
you get a segfault and your program crashes. You only get the
exception if the malloc() fails.
Now I'm not sure how windows actualy handles this but it sounds like
it allows over commit.
MfG
Goswin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-25 13:01 ` Goswin von Brederlow
@ 2013-11-25 13:55 ` Adrien Nader
2013-11-26 6:53 ` Florian Hars
2013-11-26 9:02 ` Goswin von Brederlow
0 siblings, 2 replies; 10+ messages in thread
From: Adrien Nader @ 2013-11-25 13:55 UTC (permalink / raw)
To: Goswin von Brederlow; +Cc: Amogh Margoor, caml-list
Hi,
On Mon, Nov 25, 2013, Goswin von Brederlow wrote:
> On Wed, Nov 20, 2013 at 09:56:11PM +0100, Adrien Nader wrote:
> > Hi,
> >
> > On Wed, Nov 20, 2013, Amogh Margoor wrote:
> > > Hi,
> > > The Ocaml version I am facing problem with is 3.12.1.
> > > Regards,
> > > Amogh
> > >
> > > From: Amogh Margoor
> > > Sent: Wednesday, November 20, 2013 4:19 PM
> > > To: 'caml-list@inria.fr'
> > > Subject: Ocaml crash in windows due to running out of memory
> > >
> > > Hi,
> > > We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
> > > Is that a known issue ?
> >
> > What is the exception and how much memory is your application using at
> > that moment?
>
> Many kernels only allocate address space on malloc() and assign pages
> only on use. They also support over commit of memory. Meaning
> applications can allocate more memory than the system has under the
> assumption that they won't actually use it all (or that maybe some
> memory will be freed before the apps actually use the memory).
>
> Due to that in many kernels malloc() basically never fails but when
> you use the memory and the kernel can't find any page to assign to it
> you get a segfault and your program crashes. You only get the
> exception if the malloc() fails.
>
> Now I'm not sure how windows actualy handles this but it sounds like
> it allows over commit.
As far as I know, it doesn't do overcommit. Also, I doubt that would
change anything: if you exhaust your address space, you cannot receive a
pointer to some place since it would collide with some of your already
assigned space.
On 32b, allocations have to fail once you reach 2GB of memory (or 3GB
depending on some settings). That said, such a scenario should probably
trigger an exception but so far this is just an hypothesis which cannot
be confirmed until we roughly know the memory usage.
--
Adrien Nader
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-25 13:55 ` Adrien Nader
@ 2013-11-26 6:53 ` Florian Hars
2013-11-26 9:14 ` Goswin von Brederlow
2013-11-26 9:24 ` Adrien Nader
2013-11-26 9:02 ` Goswin von Brederlow
1 sibling, 2 replies; 10+ messages in thread
From: Florian Hars @ 2013-11-26 6:53 UTC (permalink / raw)
To: caml-list
Am 25.11.2013 14:55, schrieb Adrien Nader:
> As far as I know, it doesn't do overcommit. Also, I doubt that would
> change anything: if you exhaust your address space, you cannot receive a
> pointer to some place since it would collide with some of your already
> assigned space.
That is a non-sequitur, in modern operating systems every process gets
its own space of 2^64 adresses, no matter how much physical RAM + swap
is available. And the common way to get overcommitment (at least on
unixoids, windows may be different) in't malloc anyway, it's fork, where
both processes end up with the same address space layout, but
(logically) corresponding to differen physical memory (modulo COW pages).
- Florian.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-25 13:55 ` Adrien Nader
2013-11-26 6:53 ` Florian Hars
@ 2013-11-26 9:02 ` Goswin von Brederlow
2013-11-26 9:22 ` Adrien Nader
1 sibling, 1 reply; 10+ messages in thread
From: Goswin von Brederlow @ 2013-11-26 9:02 UTC (permalink / raw)
To: caml-list
On Mon, Nov 25, 2013 at 02:55:27PM +0100, Adrien Nader wrote:
> Hi,
>
> On Mon, Nov 25, 2013, Goswin von Brederlow wrote:
> > On Wed, Nov 20, 2013 at 09:56:11PM +0100, Adrien Nader wrote:
> > > Hi,
> > >
> > > On Wed, Nov 20, 2013, Amogh Margoor wrote:
> > > > Hi,
> > > > The Ocaml version I am facing problem with is 3.12.1.
> > > > Regards,
> > > > Amogh
> > > >
> > > > From: Amogh Margoor
> > > > Sent: Wednesday, November 20, 2013 4:19 PM
> > > > To: 'caml-list@inria.fr'
> > > > Subject: Ocaml crash in windows due to running out of memory
> > > >
> > > > Hi,
> > > > We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
> > > > Is that a known issue ?
> > >
> > > What is the exception and how much memory is your application using at
> > > that moment?
> >
> > Many kernels only allocate address space on malloc() and assign pages
> > only on use. They also support over commit of memory. Meaning
> > applications can allocate more memory than the system has under the
> > assumption that they won't actually use it all (or that maybe some
> > memory will be freed before the apps actually use the memory).
> >
> > Due to that in many kernels malloc() basically never fails but when
> > you use the memory and the kernel can't find any page to assign to it
> > you get a segfault and your program crashes. You only get the
> > exception if the malloc() fails.
> >
> > Now I'm not sure how windows actualy handles this but it sounds like
> > it allows over commit.
>
> As far as I know, it doesn't do overcommit. Also, I doubt that would
> change anything: if you exhaust your address space, you cannot receive a
> pointer to some place since it would collide with some of your already
> assigned space.
> On 32b, allocations have to fail once you reach 2GB of memory (or 3GB
> depending on some settings). That said, such a scenario should probably
> trigger an exception but so far this is just an hypothesis which cannot
> be confirmed until we roughly know the memory usage.
But what if I only have 512MB ram? I can still malloc(1G), get a
pointer back and then get segfaults when I use too much of it. On
64bit systems the address space also greatly exceeds any amount of
physical ram you might have.
My reply was just an explanation of why ocaml would crash rather than
throw an out of memory exception.
MfG
Goswin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-26 6:53 ` Florian Hars
@ 2013-11-26 9:14 ` Goswin von Brederlow
2013-11-26 9:24 ` Adrien Nader
1 sibling, 0 replies; 10+ messages in thread
From: Goswin von Brederlow @ 2013-11-26 9:14 UTC (permalink / raw)
To: Florian Hars; +Cc: caml-list
On Tue, Nov 26, 2013 at 07:53:46AM +0100, Florian Hars wrote:
> Am 25.11.2013 14:55, schrieb Adrien Nader:
> >As far as I know, it doesn't do overcommit. Also, I doubt that would
> >change anything: if you exhaust your address space, you cannot receive a
> >pointer to some place since it would collide with some of your already
> >assigned space.
>
> That is a non-sequitur, in modern operating systems every process
> gets its own space of 2^64 adresses, no matter how much physical RAM
Or 2^48 on amd64 due to hardware limits. 2^47 under xen. Still plenty
more than physical ram.
> + swap is available. And the common way to get overcommitment (at
> least on unixoids, windows may be different) in't malloc anyway,
> it's fork, where both processes end up with the same address space
> layout, but (logically) corresponding to differen physical memory
> (modulo COW pages).
>
> - Florian.
Note: malloc() of larger regions usualy uses mmap(), which inside the
kernel sets up a COW region. So basically the same as fork(). Both are
causes for overcommit.
MfG
Goswin
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-26 9:02 ` Goswin von Brederlow
@ 2013-11-26 9:22 ` Adrien Nader
0 siblings, 0 replies; 10+ messages in thread
From: Adrien Nader @ 2013-11-26 9:22 UTC (permalink / raw)
To: Goswin von Brederlow; +Cc: caml-list
On Tue, Nov 26, 2013, Goswin von Brederlow wrote:
> On Mon, Nov 25, 2013 at 02:55:27PM +0100, Adrien Nader wrote:
> > Hi,
> >
> > On Mon, Nov 25, 2013, Goswin von Brederlow wrote:
> > > On Wed, Nov 20, 2013 at 09:56:11PM +0100, Adrien Nader wrote:
> > > > Hi,
> > > >
> > > > On Wed, Nov 20, 2013, Amogh Margoor wrote:
> > > > > Hi,
> > > > > The Ocaml version I am facing problem with is 3.12.1.
> > > > > Regards,
> > > > > Amogh
> > > > >
> > > > > From: Amogh Margoor
> > > > > Sent: Wednesday, November 20, 2013 4:19 PM
> > > > > To: 'caml-list@inria.fr'
> > > > > Subject: Ocaml crash in windows due to running out of memory
> > > > >
> > > > > Hi,
> > > > > We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
> > > > > Is that a known issue ?
> > > >
> > > > What is the exception and how much memory is your application using at
> > > > that moment?
> > >
> > > Many kernels only allocate address space on malloc() and assign pages
> > > only on use. They also support over commit of memory. Meaning
> > > applications can allocate more memory than the system has under the
> > > assumption that they won't actually use it all (or that maybe some
> > > memory will be freed before the apps actually use the memory).
> > >
> > > Due to that in many kernels malloc() basically never fails but when
> > > you use the memory and the kernel can't find any page to assign to it
> > > you get a segfault and your program crashes. You only get the
> > > exception if the malloc() fails.
> > >
> > > Now I'm not sure how windows actualy handles this but it sounds like
> > > it allows over commit.
> >
> > As far as I know, it doesn't do overcommit. Also, I doubt that would
> > change anything: if you exhaust your address space, you cannot receive a
> > pointer to some place since it would collide with some of your already
> > assigned space.
> > On 32b, allocations have to fail once you reach 2GB of memory (or 3GB
> > depending on some settings). That said, such a scenario should probably
> > trigger an exception but so far this is just an hypothesis which cannot
> > be confirmed until we roughly know the memory usage.
>
> But what if I only have 512MB ram? I can still malloc(1G), get a
> pointer back and then get segfaults when I use too much of it. On
> 64bit systems the address space also greatly exceeds any amount of
> physical ram you might have.
>
> My reply was just an explanation of why ocaml would crash rather than
> throw an out of memory exception.
OK.
However as far as I know and as far as I can read right now, Windows
doesn't do overcommit. Actually I'm seeing that VMWare implements its
own kind of memory overcommit to save memory on the hosts.
If you don't have enough memory, Windows (in its default configuration)
will expand its swap files. I believe that on Windows 7 the outdated
rule of swap = 2.5*RAM is still used so even with only 512MB of memory
(which is something you'll have trouble finding on Windows 7 machines)
you already get almost 2GB; and since Windows will happily expand swap
to let your machine crawl more by using more swap, I'm sure you can
almost always have 2GB usable for one process.
Another efficient way to fill up 2GB of address space is mmap() or
WindowsEquivalentEx() which are accessible through Bigarray.
In any case, it's all speculation until we get an idea of how much
memory is used by the application which crashes.
--
Adrien Nader
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Caml-list] Ocaml crash in windows due to running out of memory
2013-11-26 6:53 ` Florian Hars
2013-11-26 9:14 ` Goswin von Brederlow
@ 2013-11-26 9:24 ` Adrien Nader
1 sibling, 0 replies; 10+ messages in thread
From: Adrien Nader @ 2013-11-26 9:24 UTC (permalink / raw)
To: Florian Hars; +Cc: caml-list
On Tue, Nov 26, 2013, Florian Hars wrote:
> Am 25.11.2013 14:55, schrieb Adrien Nader:
> >As far as I know, it doesn't do overcommit. Also, I doubt that would
> >change anything: if you exhaust your address space, you cannot receive a
> >pointer to some place since it would collide with some of your already
> >assigned space.
>
> That is a non-sequitur, in modern operating systems every process
> gets its own space of 2^64 adresses, no matter how much physical RAM
> + swap is available. And the common way to get overcommitment (at
> least on unixoids, windows may be different) in't malloc anyway,
> it's fork, where both processes end up with the same address space
> layout, but (logically) corresponding to differen physical memory
> (modulo COW pages).
I probably didn't make that clear: I'm only talking about 32 bit
processes. These are still very common on Windows.
Btw, you don't have fork() on Windows. Memory sharing is not done in the
same way at all.
--
Adrien Nader
^ permalink raw reply [flat|nested] 10+ messages in thread
* [Caml-list] Ocaml crash in windows due to running out of memory
@ 2013-11-20 10:48 Amogh Margoor
0 siblings, 0 replies; 10+ messages in thread
From: Amogh Margoor @ 2013-11-20 10:48 UTC (permalink / raw)
To: caml-list
[-- Attachment #1: Type: text/plain, Size: 193 bytes --]
Hi,
We have memory intensive application coded in Ocaml, which crashes on Windows 7 when it runs out of memory instead of throwing an exception.
Is that a known issue ?
Regards,
Amogh
[-- Attachment #2: Type: text/html, Size: 2156 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-11-26 9:24 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-11-20 10:52 [Caml-list] Ocaml crash in windows due to running out of memory Amogh Margoor
2013-11-20 20:56 ` Adrien Nader
2013-11-25 13:01 ` Goswin von Brederlow
2013-11-25 13:55 ` Adrien Nader
2013-11-26 6:53 ` Florian Hars
2013-11-26 9:14 ` Goswin von Brederlow
2013-11-26 9:24 ` Adrien Nader
2013-11-26 9:02 ` Goswin von Brederlow
2013-11-26 9:22 ` Adrien Nader
-- strict thread matches above, loose matches on Subject: below --
2013-11-20 10:48 Amogh Margoor
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).