From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 8E65C7F98F for ; Thu, 10 Aug 2017 15:12:07 +0200 (CEST) Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=lwhite@janestreet.com; spf=Pass smtp.mailfrom=lwhite@janestreet.com; spf=None smtp.helo=postmaster@mxout1.mail.janestreet.com Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of lwhite@janestreet.com) identity=pra; client-ip=38.105.200.78; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="lwhite@janestreet.com"; x-sender="lwhite@janestreet.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of lwhite@janestreet.com designates 38.105.200.78 as permitted sender) identity=mailfrom; client-ip=38.105.200.78; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="lwhite@janestreet.com"; x-sender="lwhite@janestreet.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mxout1.mail.janestreet.com) identity=helo; client-ip=38.105.200.78; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="lwhite@janestreet.com"; x-sender="postmaster@mxout1.mail.janestreet.com"; x-conformance=sidf_compatible IronPort-PHdr: =?us-ascii?q?9a23=3AmVMwxRUzQONC375ZKNmqTl+V+bjV8LGtZVwlr6E/?= =?us-ascii?q?grcLSJyIuqrYZhSPt8tkgFKBZ4jH8fUM07OQ6P+wHzFYqb+681k8M7V0Hycfjs?= =?us-ascii?q?sXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aMlzFOAF0?= =?us-ascii?q?PuX4HJLJx4Tyjrjqus6bXwIdpze7Z75uLF2foQzU/uwXhY9vMO5lyRbPpHZUe+?= =?us-ascii?q?1azGZtJFaXkgzU6cK5/Zol+CNV7aEP7clFBIH3eOwHTb1EAXxyN3815dHmnRvK?= =?us-ascii?q?SwaU+mERX3lQmR1NVVuWpCrmV4v853Op/tF23zOXaIivFeg5?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0BhAQCFWoxZfU7IaSZcHAEBBAEBCgEBF?= =?us-ascii?q?gEBAQMBAQEJAQEBgkQ+giUHhSWYa4FPH4g2j3GFRwKEewdCFQEBAQEBAQEBAQE?= =?us-ascii?q?BEgEBCRYIV4IzBQEeAQWCPAECAgEjHQEBNwEECwsEBzcCAiEBEgEFARwGEwiKD?= =?us-ascii?q?wMNCAOgYz+LH2uCJoMIAQEFhCgNhCUBAQEBAQEEAQEBAQEBAQEYCBKDFoICgUy?= =?us-ascii?q?FCoJXhS+CYZ9mPI9EhHSLQocQjC+IFxQfgRU1gSxTJBVJGgaEaA8cgWh1iFGBU?= =?us-ascii?q?wEBAQ?= X-IPAS-Result: =?us-ascii?q?A0BhAQCFWoxZfU7IaSZcHAEBBAEBCgEBFgEBAQMBAQEJAQE?= =?us-ascii?q?BgkQ+giUHhSWYa4FPH4g2j3GFRwKEewdCFQEBAQEBAQEBAQEBEgEBCRYIV4IzB?= =?us-ascii?q?QEeAQWCPAECAgEjHQEBNwEECwsEBzcCAiEBEgEFARwGEwiKDwMNCAOgYz+LH2u?= =?us-ascii?q?CJoMIAQEFhCgNhCUBAQEBAQEEAQEBAQEBAQEYCBKDFoICgUyFCoJXhS+CYZ9mP?= =?us-ascii?q?I9EhHSLQocQjC+IFxQfgRU1gSxTJBVJGgaEaA8cgWh1iFGBUwEBAQ?= X-IronPort-AV: E=Sophos;i="5.41,353,1498514400"; d="scan'208,217";a="286683928" Received: from mxout1.mail.janestreet.com ([38.105.200.78]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 10 Aug 2017 15:12:06 +0200 Received: from [172.27.56.106] (helo=tot-qpr-mailcore2) by mxout1.mail.janestreet.com with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1dfnFt-0006ya-Dc for caml-list@inria.fr; Thu, 10 Aug 2017 09:12:05 -0400 X-JS-Flow: external X-JS-Scanner-attachment: (ok) No attachments Received: by tot-qpr-mailcore2 with ocaml/mailcore/mailcore 1.0+108 (0e74f06049cc) (envelope-from ) id BZjFul-4LJreA-Mw; 2017-08-10 09:12:05.412523-04:00 Received: from mail-io0-f200.google.com ([209.85.223.200]) by mxgoog1.mail.janestreet.com with esmtps (TLSv1.2:ECDHE-RSA-AES128-GCM-SHA256:128) (Exim 4.89) (envelope-from ) id 1dfnFt-0003rJ-CB for caml-list@inria.fr; Thu, 10 Aug 2017 09:12:05 -0400 Received: by mail-io0-f200.google.com with SMTP id s23so20292477ioe.8 for ; Thu, 10 Aug 2017 06:12:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ci2TlsGFo64LtkBApZZhOtIGJUtdAQ4Bgn7wP3RebLI=; b=eE5iBfz6iAstEUUkI59TONTSQrA60ndsHCYofOpLkHPPq3F+kPUncXLXFtDE2ZyvMf sJUPkFdmXWMgJZ320XNdRaNL4cEBD3D8SWkSdKK5KhTwgmfIAcxF/U8HYTUsmqpC/Mi8 Ti12mrjMDYKVbS6YC8kLz7Ur0qZzgPyOXoXLE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ci2TlsGFo64LtkBApZZhOtIGJUtdAQ4Bgn7wP3RebLI=; b=dxUIM12PiFKTuL8KGF5tT+uxVDGqKTgqWhina6X9D/LoLR6RQBFhYdO/PH/jM1RAoS jCBXU5z1AdZPCQjhvBoFUkQSBXvlWFYxwIfzdgpTQzQh8G+4feS6TbefvZkyo46sqiFQ NBdtUJecXRnF7iQ3gJFDvK/fZGmCyAZNDKlzqZKLqThyTBrhW+O1u7ztfhZgltz0pQc6 yrdVemzBaA+XsqbbzxK79LgHRIb1OWkJbILbgcBQufGAWD1MaGc87mdfJ+5zMaL2lX3D ZS1SYkzlZp+aSk4bEMWLp9OWAGXed57xHGoL5t0Zi1wCMf+WTqARrCT9EiBoiq+litzf OvjA== X-Gm-Message-State: AIVw111sV/wykbV1TXjhxKDIHa3idNxre2irBi0jynWUHGaZxsmcfA6Q 1r2AFIJAvCOiSrqffqI7dN8BLqQT26egmKcqtHVjjXE6byThGV5lugz4Hq4VbBimKpJBSEnLfHQ 1nT830iDtEbO53erl X-Received: by 10.36.77.134 with SMTP id l128mr9244763itb.44.1502370724593; Thu, 10 Aug 2017 06:12:04 -0700 (PDT) X-Received: by 10.36.77.134 with SMTP id l128mr9244751itb.44.1502370724432; Thu, 10 Aug 2017 06:12:04 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.145.212 with HTTP; Thu, 10 Aug 2017 06:12:03 -0700 (PDT) In-Reply-To: References: From: Leo White Date: Thu, 10 Aug 2017 14:12:03 +0100 Message-ID: To: Kenneth Adam Miller Cc: caml users Content-Type: multipart/alternative; boundary="001a114497b259ef56055665f1d5" X-JS-Exim-Data-Received: 2017-08-10 09:12:05-0400 X-JS-Processed-by: mailcore Subject: Re: [Caml-list] Overflow with Spacetime --001a114497b259ef56055665f1d5 Content-Type: text/plain; charset="UTF-8" > Possibly I ran into a space leak while trying to find a space leak. 7GB for a 900MB profile sounds about right unfortunately. The format of spacetime profiles makes it difficult for the viewer to do much better than that. I hope to get around to changing the format to fix this, but probably won't until some time next year. I would recommend trying to get a smaller profile. Taking snapshots less often will help, as will running the program for a shorter time (even with the same number of snapshots). Failing that try getting access to a machine with more memory. On 9 August 2017 at 08:37, Kenneth Adam Miller wrote: > Hello all, > > > I wrote a particular application, and after profiling I can see that the > garbage collector is using 95% of the time in the application and that it > uses up to 6 gigs of memory on a 1 megabyte input file for a run that > totals out to be about 2 minutes. A 27 kilobyte input file takes seconds to > run, so I know that there is some explosive leak or otherwise terrible > issue with the space complexity here. I doubled down, and got compilation > to run to completion using _tags to make sure that there were symbols and > profiling turned on (thanks Ivan, Drup). > > Additionally, I compiled my application with spacetime and ran it to > obtain a `spacetime-` binary, which I then attempted to get a report > on with prof_spacetime. prof_spacetime went up to about 7.3 gigs in memory > usage before it repeatedly would die with `Killed` or `Stack overflow` as > the error. I'm trying to figure out what else there is I can do to make > sure that the process runs to completion. I double checked to make sure > that I was running the prof_spacetime command built with just a 4.04.0 > compiler to ensure that it wouldn't profile itself while trying to "Process > series". My spacetime dump is about 900 Mb, so I understand that it would > use some memory processing it. Possibly I ran into a space leak while > trying to find a space leak. > > In the meantime, I will try and find some intermediate size input to get > something I can work on - thought I should report this. > > > --001a114497b259ef56055665f1d5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
>=C2=A0Possibly I ran = into a space leak while trying to find a space leak.

7GB for a 900MB profile sounds about right unfortunately. The format of s= pacetime profiles makes it difficult for the viewer to do much better than = that. I hope to get around to changing the format to fix this, but probably= won't until some time next year.

I would= recommend trying to get a smaller profile. Taking snapshots less often wil= l help, as will running the program for a shorter time (even with the same = number of snapshots). Failing that try getting access to a machine with mor= e memory.

On 9 August 2017 at 08:37, Kenneth Adam Miller <kenne= thadammiller@gmail.com> wrote:
Hello all,


I wrote a= particular application, and after profiling I can see that the garbage col= lector is using 95% of the time in the application and that it uses up to 6= gigs of memory on a 1 megabyte input file for a run that totals out to be = about 2 minutes. A 27 kilobyte input file takes seconds to run, so I know t= hat there is some explosive leak or otherwise terrible issue with the space= complexity here. I doubled down, and got compilation to run to completion = using _tags to make sure that there were symbols and profiling turned on (t= hanks Ivan, Drup).=C2=A0

Additionally, I compiled my application wit= h spacetime and ran it to obtain a `spacetime-<pid>` binary, which I = then attempted to get a report on with prof_spacetime. prof_spacetime went = up to about 7.3 gigs in memory usage before it repeatedly would die with `K= illed` or `Stack overflow` as the error. I'm trying to figure out what = else there is I can do to make sure that the process runs to completion. I = double checked to make sure that I was running the prof_spacetime command b= uilt with just a 4.04.0 compiler to ensure that it wouldn't profile its= elf while trying to "Process series". My spacetime dump is about = 900 Mb, so I understand that it would use some memory processing it. Possib= ly I ran into a space leak while trying to find a space leak.=C2=A0

= In the meantime, I will try and find some intermediate size input to get so= mething I can work on - thought I should report this.

<= div>

--001a114497b259ef56055665f1d5--