[Caml-list] Overflow with Spacetime

[Caml-list] Overflow with Spacetime

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]

Hello all,


I wrote a particular application, and after profiling I can see that the
garbage collector is using 95% of the time in the application and that it
uses up to 6 gigs of memory on a 1 megabyte input file for a run that
totals out to be about 2 minutes. A 27 kilobyte input file takes seconds to
run, so I know that there is some explosive leak or otherwise terrible
issue with the space complexity here. I doubled down, and got compilation
to run to completion using _tags to make sure that there were symbols and
profiling turned on (thanks Ivan, Drup).

Additionally, I compiled my application with spacetime and ran it to obtain
a `spacetime-<pid>` binary, which I then attempted to get a report on with
prof_spacetime. prof_spacetime went up to about 7.3 gigs in memory usage
before it repeatedly would die with `Killed` or `Stack overflow` as the
error. I'm trying to figure out what else there is I can do to make sure
that the process runs to completion. I double checked to make sure that I
was running the prof_spacetime command built with just a 4.04.0 compiler to
ensure that it wouldn't profile itself while trying to "Process series". My
spacetime dump is about 900 Mb, so I understand that it would use some
memory processing it. Possibly I ran into a space leak while trying to find
a space leak.

In the meantime, I will try and find some intermediate size input to get
something I can work on - thought I should report this.

[-- Attachment #2: Type: text/html, Size: 1583 bytes --]

Re: [Caml-list] Overflow with Spacetime

[Leo White]

[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]

> Possibly I ran into a space leak while trying to find a space leak.

7GB for a 900MB profile sounds about right unfortunately. The format of
spacetime profiles makes it difficult for the viewer to do much better than
that. I hope to get around to changing the format to fix this, but probably
won't until some time next year.

I would recommend trying to get a smaller profile. Taking snapshots less
often will help, as will running the program for a shorter time (even with
the same number of snapshots). Failing that try getting access to a machine
with more memory.

On 9 August 2017 at 08:37, Kenneth Adam Miller <kennethadammiller@gmail.com>
wrote:

> Hello all,
>
>
> I wrote a particular application, and after profiling I can see that the
> garbage collector is using 95% of the time in the application and that it
> uses up to 6 gigs of memory on a 1 megabyte input file for a run that
> totals out to be about 2 minutes. A 27 kilobyte input file takes seconds to
> run, so I know that there is some explosive leak or otherwise terrible
> issue with the space complexity here. I doubled down, and got compilation
> to run to completion using _tags to make sure that there were symbols and
> profiling turned on (thanks Ivan, Drup).
>
> Additionally, I compiled my application with spacetime and ran it to
> obtain a `spacetime-<pid>` binary, which I then attempted to get a report
> on with prof_spacetime. prof_spacetime went up to about 7.3 gigs in memory
> usage before it repeatedly would die with `Killed` or `Stack overflow` as
> the error. I'm trying to figure out what else there is I can do to make
> sure that the process runs to completion. I double checked to make sure
> that I was running the prof_spacetime command built with just a 4.04.0
> compiler to ensure that it wouldn't profile itself while trying to "Process
> series". My spacetime dump is about 900 Mb, so I understand that it would
> use some memory processing it. Possibly I ran into a space leak while
> trying to find a space leak.
>
> In the meantime, I will try and find some intermediate size input to get
> something I can work on - thought I should report this.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2788 bytes --]

Re: [Caml-list] Overflow with Spacetime

[Kenneth Adam Miller]

[-- Attachment #1: Type: text/plain, Size: 2372 bytes --]

Ok, that's understandable. No complaints on my end, spacetime is awesome.

On Thu, Aug 10, 2017 at 9:12 AM, Leo White <lwhite@janestreet.com> wrote:

> > Possibly I ran into a space leak while trying to find a space leak.
>
> 7GB for a 900MB profile sounds about right unfortunately. The format of
> spacetime profiles makes it difficult for the viewer to do much better than
> that. I hope to get around to changing the format to fix this, but probably
> won't until some time next year.
>
> I would recommend trying to get a smaller profile. Taking snapshots less
> often will help, as will running the program for a shorter time (even with
> the same number of snapshots). Failing that try getting access to a machine
> with more memory.
>
> On 9 August 2017 at 08:37, Kenneth Adam Miller <
> kennethadammiller@gmail.com> wrote:
>
>> Hello all,
>>
>>
>> I wrote a particular application, and after profiling I can see that the
>> garbage collector is using 95% of the time in the application and that it
>> uses up to 6 gigs of memory on a 1 megabyte input file for a run that
>> totals out to be about 2 minutes. A 27 kilobyte input file takes seconds to
>> run, so I know that there is some explosive leak or otherwise terrible
>> issue with the space complexity here. I doubled down, and got compilation
>> to run to completion using _tags to make sure that there were symbols and
>> profiling turned on (thanks Ivan, Drup).
>>
>> Additionally, I compiled my application with spacetime and ran it to
>> obtain a `spacetime-<pid>` binary, which I then attempted to get a report
>> on with prof_spacetime. prof_spacetime went up to about 7.3 gigs in memory
>> usage before it repeatedly would die with `Killed` or `Stack overflow` as
>> the error. I'm trying to figure out what else there is I can do to make
>> sure that the process runs to completion. I double checked to make sure
>> that I was running the prof_spacetime command built with just a 4.04.0
>> compiler to ensure that it wouldn't profile itself while trying to "Process
>> series". My spacetime dump is about 900 Mb, so I understand that it would
>> use some memory processing it. Possibly I ran into a space leak while
>> trying to find a space leak.
>>
>> In the meantime, I will try and find some intermediate size input to get
>> something I can work on - thought I should report this.
>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 3308 bytes --]