From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by c5ff346549e7 (Postfix) with ESMTPS id D3E245D5 for ; Wed, 29 May 2019 15:22:43 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.60,527,1549926000"; d="scan'208,217";a="385268708" Received: from sympa.inria.fr ([193.51.193.213]) by mail2-relais-roc.national.inria.fr with ESMTP; 29 May 2019 17:22:42 +0200 Received: by sympa.inria.fr (Postfix, from userid 20132) id 2B70F826E1; Wed, 29 May 2019 17:22:42 +0200 (CEST) Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 77F4B7ED69 for ; Wed, 29 May 2019 17:22:30 +0200 (CEST) Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=jf451@cam.ac.uk; spf=Pass smtp.mailfrom=jf451@cam.ac.uk; spf=Pass smtp.helo=postmaster@ppsw-41.csi.cam.ac.uk IronPort-PHdr: =?us-ascii?q?9a23=3AnCK03h2yVjGuX0masmDT+DRfVm0co7zxezQtwd8Z?= =?us-ascii?q?sesWIvzxwZ3uMQTl6Ol3ixeRBMOHsqsC0rCJ+PC/EUU7or+5+EgYd5JNUxJXwe?= =?us-ascii?q?43pCcHRPC/NEvgMfTxZDY7FskRHHVs/nW8LFQHUJ2mPw6arXK99yMdFQviPgRp?= =?us-ascii?q?OOv1BpTSj8Oq3Oyu5pHfeQpFiCegbb9oMRm6swfcusYVjIZgN6081gbHrnxUdu?= =?us-ascii?q?pM2GhmP0iTnxHy5sex+J5s7SFdsO8/+sBDTKv3Yb02QaRXAzo6PW814tbrtQTY?= =?us-ascii?q?QguU+nQcSGQWnQFWDAXD8Rr3Q43+sir+tup6xSmaIcj7Rq06VDi+86tmTgLjhT?= =?us-ascii?q?wZPDAl7m7Yls1wjLpaoB2/oRx/35XUa5yROPZnY6/RYc8WSW9HU81MVSJOH5m8?= =?us-ascii?q?YpMIAeUPMulWrIvyp1UAoxW+GweiGf/gxyRSiXPqx6A3yf4sHBvE0QEmAtkAsG?= =?us-ascii?q?7UrNLwNKoKX+y7yK7IzTPeZP1Wwzfy9o7IfQwhof2CQLl9dsjRyUcgGg7Fk1md?= =?us-ascii?q?spDqMCmQ1ugXqWeU8/BsVf+si2M+rQx6vzahxsApiobTh4IVzEjJ9Sp4wIYpJd?= =?us-ascii?q?24VVV0bcS4H5tXsiGWL4R2QsI+Q2FopSY10acKtYSncygNzZQqwQPUZf+fc4WQ?= =?us-ascii?q?/x7uWvudLS1liH54Zb6znRW//VK9xuDzS8W4yFdHoytfntXRq3wBygbf58edRv?= =?us-ascii?q?dj4Eus2zCC3B3J5O5eO0A7j6/bJoYhwrEukpoTtlzOHjfumEXtgq6ab0op9vWy?= =?us-ascii?q?5+v7ebXmp4WQOJNuhQH7KKghgNCwDf4lMggNR2Sb+OK826P//UDhXblHgOA6nr?= =?us-ascii?q?PEvJzHOMgXvK20DxVI3oss9hqzFzKm384ZnXkDIlJFYhWHj43xNlHMLvD1Avey?= =?us-ascii?q?j0m3nTh33f/GO6ftDY/RIXTZjbfhfq5x61RAxwor0dBf+5VUB6kdL/3pX0/xsM?= =?us-ascii?q?XUDhs4Mwyv3+bqE85914MbWWKXGKCVKqLSsVmS5uIuOeaAfoEVuCzlJ/gg/fHu?= =?us-ascii?q?jHs5mVEHfamu2JsacHe4HvZgI0WeZnbsh8kOEXwPvgoiVOzlkkCCUSJTZ3aqQa?= =?us-ascii?q?08/Co7CIWgDYjZQoCtgaCB3SeiEpBVaW1LCVGBHWnoeoiLVPoAcT+eL8t9njAa?= =?us-ascii?q?V7WsSoss2BC3uA/4xbpqIerZ9jAAuJ/7yNd6/ejTmQso+jNoFcidzmKNQnp6nm?= =?us-ascii?q?wSXD82wKV/rlZ8yleHy6R3n/tYFdlV6vhUTAo6MYPcz/dmC9/sQALPY9aJSVe4?= =?us-ascii?q?Tdi+HT1iBu42lpUgakxnGt6vxjTOlwSnGKQckaDBTMg6+6jG3nP8YcJw/HjLz7?= =?us-ascii?q?IoiUUORcBGMGm+nKk5/A/WUcqB2WSHnqDiWqMA2zDG9Gaf1iDG6EBGXyZxXKjI?= =?us-ascii?q?G3cFaR2Fg87+4xaIbbioQZo9Pw1KyYTKfqlENoCwpV5PQbHqM5LDYDTiyC+LGR?= =?us-ascii?q?+Uy+bUP8LRcGIH0XCYURBcylFBzTO9LQE7QxyZjSfbBT1qG0joZhq1o+J3rTWy?= =?us-ascii?q?RQkpzFPTNhAz5/+O4hcQwMekZbYT07YD4nxzsy1vAxPhhpTdENvGrANkOqxXJ8?= =?us-ascii?q?4+sg4eiTDp8jdlN5nlFJhMw0YEel0u7Ujn0lN+AcNdkppyoQ=3D=3D?= X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0BjAgBPo+5ch40Ib4NlHAEBAQQBAQcEA?= =?us-ascii?q?QGBZYJ6UwEwKIQTgV+HHIpbAQEGgTV+iFKRBgEIAQMBDCMMAQGDCYE3AoJ1HAY?= =?us-ascii?q?BBDQTAQMBAQQBAQIBAQMBEwEBAQgNCQgpIwyCOikBgmcBAgMjHQEBNwEPCxgqA?= =?us-ascii?q?gIsKwYBEoMiAYF2FA+ob3GBL4J5AQEFgTIBgQmCfiGBTgMGgTSCFoRnhWWBB4E?= =?us-ascii?q?RJ4I2By4+gmECgg6CXYJYgTEBAQGmRmkBBgIBAoINX4VZjGEhgh+KaolKjHWFW?= =?us-ascii?q?4ErjyOBZoF4gQE/LhpWgU6CDxqDVod6glo+AQExjkcBAQ?= X-IPAS-Result: =?us-ascii?q?A0BjAgBPo+5ch40Ib4NlHAEBAQQBAQcEAQGBZYJ6UwEwKIQ?= =?us-ascii?q?TgV+HHIpbAQEGgTV+iFKRBgEIAQMBDCMMAQGDCYE3AoJ1HAYBBDQTAQMBAQQBA?= =?us-ascii?q?QIBAQMBEwEBAQgNCQgpIwyCOikBgmcBAgMjHQEBNwEPCxgqAgIsKwYBEoMiAYF?= =?us-ascii?q?2FA+ob3GBL4J5AQEFgTIBgQmCfiGBTgMGgTSCFoRnhWWBB4ERJ4I2By4+gmECg?= =?us-ascii?q?g6CXYJYgTEBAQGmRmkBBgIBAoINX4VZjGEhgh+KaolKjHWFW4ErjyOBZoF4gQE?= =?us-ascii?q?/LhpWgU6CDxqDVod6glo+AQExjkcBAQ?= X-IronPort-AV: E=Sophos;i="5.60,527,1549926000"; d="scan'208,217";a="385268575" X-MGA-submission: =?us-ascii?q?MDEkXopC2iXhbKPGX49nrbun0Hm3Mgy9KhDoQS?= =?us-ascii?q?OEhG3CCXvIT97JXgAVjHsfgz5ZRWgoCgtLJ3Mhn9L+LJG4S/qM/K97Ft?= =?us-ascii?q?jGsshlyudlqPGEsk2Fi0hRsx4LYxGCsaWKoePePtpVOp+KZmpHQy0Czg?= =?us-ascii?q?cYAdeVH3PaPs8/J45LsO10Yw=3D=3D?= Received: from ppsw-41.csi.cam.ac.uk ([131.111.8.141]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 May 2019 17:22:18 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cam.ac.uk; s=20180806.ppsw; h=Content-Type:Subject:Cc:To:From:Date:References: In-Reply-To:Message-Id:Mime-Version:Sender:Reply-To:Content-Transfer-Encoding :Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=CWDwk+FmGtiEgm4uoL4nAG6szwCrC8TYGmvcaAxuBHE=; b=hvqVC0cXQWz9IS3Hf8MwPGkhgM 0w42COQN0AhfYNSJmHMOs5fXOxLsBDoCTPalUFLFB6hnieQtxHewSsy4CcvPb1/bIB+QNjpEp0Ehr qTn28RzXgrysl+vPZM3o/JM8W19shs9J8mTEEXFl8V8OodagA+RVGUwIX1scwqoHMMF0=; X-Cam-AntiVirus: no malware found X-Cam-ScannerInfo: http://help.uis.cam.ac.uk/email-scanner-virus Received: from auth1-smtp.messagingengine.com ([66.111.4.227]:38875) by ppsw-41.csi.cam.ac.uk (smtp.hermes.cam.ac.uk [131.111.8.159]:587) with esmtpsa (LOGIN:jf451) (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) id 1hW0PA-0017GN-QY (Exim 4.91) (return-path ); Wed, 29 May 2019 16:22:16 +0100 Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailauth.nyi.internal (Postfix) with ESMTP id 5C35721FF5; Wed, 29 May 2019 11:22:14 -0400 (EDT) Received: from imap2 ([10.202.2.52]) by compute6.internal (MEProxy); Wed, 29 May 2019 11:22:14 -0400 X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddvjedgkeelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepofgfggfkjghffffhvffutgesrgdtreerreerjeenucfhrhhomhepfdflohhn ucfhrhgvnhgthhdfuceojhhfgeehudestggrmhdrrggtrdhukheqnecuffhomhgrihhnpe hgihhthhhusgdrtghomhenucfrrghrrghmpehmrghilhhfrhhomhepohhjnhhoodhmvghs mhhtphgruhhthhhpvghrshhonhgrlhhithihqdekgeekiedvheeggedqudeltddvjeeile eiqdhjfheghedupeeptggrmhdrrggtrdhukhesfhgrshhtmhgrihhlrdgtohhmnecuvehl uhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id AD64DE00A1; Wed, 29 May 2019 11:22:13 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.1.6-555-g49357e1-fmstable-20190528v2 Mime-Version: 1.0 Message-Id: In-Reply-To: References: <8d058e9d-5293-46f1-9941-7c33fd232e04@www.fastmail.com> Date: Wed, 29 May 2019 16:22:13 +0100 From: "Jon French" To: "Fabrice Le Fessant" , "Ivan Gotovchits" Cc: caml-list Content-Type: multipart/alternative; boundary=14104797783642638de8b435ec21f658 Subject: Re: [Caml-list] Only bytecode version of executable allocating huge amounts Reply-To: "Jon French" X-Loop: caml-list@inria.fr X-Sequence: 17589 Errors-to: caml-list-owner@inria.fr Precedence: list Precedence: bulk Sender: caml-list-request@inria.fr X-no-archive: yes List-Id: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --14104797783642638de8b435ec21f658 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable Thanks all for your replies. I've opened a github issue here: https://gi= thub.com/ocaml/ocaml/issues/8699 On Tue, 28 May 2019, at 18:58, Fabrice Le Fessant wrote: > A common difference between bytecode and native programs is that the n= ative compiler computes the liveness of variables in the stack, so that = they are not scanned anymore afterwards (or coalesced with other variabl= es). For example, if you have a data structure that is allocated before = a computation, but not used in the computation, the structure might be k= ept alive in the bytecode, but garbage collected in native code. >=20 > Le mar. 28 mai 2019 =C3=A0 19:23, Ivan Gotovchits a =C3= =A9crit : >> My (common) suspect is Zarith. It uses a different backend when compi= led to bytecode, so I would start my investigation from this point :)=20= >>=20 >> Hope it helps, >> Ivan >>=20 >> On Tue, May 28, 2019 at 8:27 AM Jon French wrote: >>> __ >>> Hi all, >>>=20 >>> I'm wondering if anyone on this list might have encountered this iss= ue before, since it's got me pretty stumped. >>>=20 >>> We have an OCaml project ( https://github.com/rems-project/sail ) wh= ich tries to allocate huge amounts of memory, but only when compiled to = bytecode. The native version works perfectly fine. >>>=20 >>> strace for an example run ends in: >>>=20 >>>> openat(AT_FDCWD, "/home/ojno/work/sail2/lib/vector_dec.sail", O_RDO= NLY|O_CLOEXEC) =3D 3 >>>> lseek(3, 0, SEEK_CUR) =3D 0 >>>> read(3, "$ifndef _VECTOR_DEC\n$define _VEC"..., 65536) =3D 7031 >>>> mmap(NULL, 1965819695104, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANO= NYMOUS, -1, 0) =3D -1 ENOMEM (Cannot allocate memory) >>>> brk(0x5729f32c4000) =3D 0x55603f2f4000 >>>> mmap(NULL, 1965819826176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANO= NYMOUS, -1, 0) =3D -1 ENOMEM (Cannot allocate memory) >>>> write(2, "Fatal error: out of memory.\n", 28Fatal error: out of mem= ory. >>>> ) =3D 28 >>>> exit_group(2) =3D ? >>>> +++ exited with 2 +++ >>>=20 >>> which naturally fails since I don't have 2 TB of memory available.=20= >>>=20 >>> The exact amount of the allocation varies from input to input and ev= en run to run, but is always that approximate size. There's no character= istic pattern of heap or stack running out of control if I turn on memor= y debugging info in $OCAMLRUNPARAM, just that one huge allocation. When = run in ocamldebug, it also doesn't appear to be failing at a location wh= ich is sensible to be allocating such amounts of memory -- and the locat= ion also varies with the input. And on tiny inputs it doesn't seem to tr= y and make the allocation at all. >>>=20 >>> Are we actually seeing a compiler/runtime bug here? Is there somethi= ng I'm missing? >>>=20 >>> Thanks, >>>=20 >>> Jon French >>> Computer Laboratory >>> University of Cambridge > --=20 > Fabrice LE FESSANT > CEO, OCamlPro SAS --14104797783642638de8b435ec21f658 Content-Type: text/html;charset=utf-8 Content-Transfer-Encoding: quoted-printable
Thanks all for = your replies. I've opened a github issue here: https://github.com/ocaml/ocaml/issues/8699=

On Tue, 28 May 2019, at 18:58, Fabrice= Le Fessant wrote:
A common difference between bytecode and native programs = is that the native compiler computes the liveness of variables in the st= ack, so that they are not scanned anymore afterwards (or coalesced with = other variables). For example, if you have a data structure that is allo= cated before a computation, but not used in the computation, the structu= re might be kept alive in the bytecode, but garbage collected in native = code.
On Tue, May 28, 2019 at 8:27 AM Jon French <jf451@cam.ac.uk> wrote:

=
Hi all,

I'm wondering if an= yone on this list might have encountered this issue before, since it's g= ot me pretty stumped.

We have an OCaml proj= ect ( https://github.co= m/rems-project/sail ) which tries to allocate huge amounts of memory= , but only when compiled to bytecode. The native version works perfectly= fine.

strace for an example run ends in:

openat(AT_FDCWD, "= /home/ojno/work/sail2/lib/vector_dec.sail", O_RDONLY|O_CLOEXEC) =3D 3
lseek(3, 0, SEEK_CUR)      &nb= sp;            =3D= 0
read(3, "$ifndef _VECTOR_DEC\n$define _VEC"..., 65536) = =3D 7031
mmap(NULL, 1965819695104, PROT_READ|PROT_WRITE, M= AP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D -1 ENOMEM (Cannot allocate memory)<= br>
brk(0x5729f32c4000)      &nb= sp;           &nb= sp;  =3D 0x55603f2f4000
mmap(NULL, 1965819826176, PRO= T_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =3D -1 ENOMEM (Cann= ot allocate memory)
write(2, "Fatal error: out of memory.\= n", 28Fatal error: out of memory.
) =3D 28
e= xit_group(2)          =             =      =3D ?
+++ exited with 2 +++

which naturally fails since I don't= have 2 TB of memory available.

The exact = amount of the allocation varies from input to input and even run to run,= but is always that approximate size. There's no characteristic pattern = of heap or stack running out of control if I turn on memory debugging in= fo in $OCAMLRUNPARAM, just that one huge allocation. When run in ocamlde= bug, it also doesn't appear to be failing at a location which is sensibl= e to be allocating such amounts of memory -- and the location also varie= s with the input. And on tiny inputs it doesn't seem to try and make the= allocation at all.

Are we actually seeing = a compiler/runtime bug here? Is there something I'm missing?

Thanks,

Jon French
Computer Laboratory
University of Cambridge
--
<= div class=3D"qt-gmail_signature" dir=3D"ltr">
Fabrice LE FESSANT
=
CEO, OCamlPro SAS

= --14104797783642638de8b435ec21f658--