From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 682657FEEB for ; Sat, 18 Jun 2016 17:11:51 +0200 (CEST) IronPort-PHdr: 9a23:RwL5+B2iPCbFPejrsmDT+DRfVm0co7zxezQtwd8ZsekWKfad9pjvdHbS+e9qxAeQG96LurQV0aGJ7ejJYi8p39WoiDg6aptCVhsI2409vjcLJ4q7M3D9N+PgdCcgHc5PBxdP9nC/NlVJSo6lPwWB6kO74TNaIBjjLw09fr2zQd6DyZXtnL/ss7ToICx2xxOFKYtoKxu3qQiD/uI3uqBFbpgL9x3Sv3FTcP5Xz247bXianhL7+9vitMU7q3cYk7sb+sVBSaT3ebgjBfwdVWx+cjMD39DwrRTIUSeI43IdVC1WzksJUED560TGwI1vsRzXuvV83mHOMMHpTLZ3XDDn6KxiTRvAhTsALTk6tmfalpojorhcpUfrghVl34/SV7vTA9xzY6NRYGQXXyAJCt5WTDBpB4qmaYYSSeEGOLAL/MHGu1ISoE7mVkGXD+T1x2oN2yb7 Authentication-Results: mail2-smtp-roc.national.inria.fr; spf=None smtp.pra=post@marneu.com; spf=Pass smtp.mailfrom=post@marneu.com; spf=None smtp.helo=postmaster@zulu1878.startdedicated.de Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of post@marneu.com) identity=pra; client-ip=62.75.236.16; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="post@marneu.com"; x-sender="post@marneu.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail2-smtp-roc.national.inria.fr: domain of post@marneu.com designates 62.75.236.16 as permitted sender) identity=mailfrom; client-ip=62.75.236.16; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="post@marneu.com"; x-sender="post@marneu.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@zulu1878.startdedicated.de) identity=helo; client-ip=62.75.236.16; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="post@marneu.com"; x-sender="postmaster@zulu1878.startdedicated.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0B0BAAJZGVX/xDsSz5dhBRLATGCfqZKkRyBeiJghRUCgVcUAQEBAQEBAQFkJ4IxghoBAQEDAQwXBC4ZCxAJAhgNHQICRRIGExKIBAMPDAEJkkudHYwwA4NoAQEBAQEFAQEBAQEBARIOhS+Cb4JWhCyDFSuCLwWGSwySHwGDLYJYiCSCN4cKhWGPdQIeNoIHHReBN2yJBYFDAQEB X-IPAS-Result: A0B0BAAJZGVX/xDsSz5dhBRLATGCfqZKkRyBeiJghRUCgVcUAQEBAQEBAQFkJ4IxghoBAQEDAQwXBC4ZCxAJAhgNHQICRRIGExKIBAMPDAEJkkudHYwwA4NoAQEBAQEFAQEBAQEBARIOhS+Cb4JWhCyDFSuCLwWGSwySHwGDLYJYiCSCN4cKhWGPdQIeNoIHHReBN2yJBYFDAQEB X-IronPort-AV: E=Sophos;i="5.26,487,1459807200"; d="asc'?scan'208";a="222963168" Received: from wanjo.de (HELO zulu1878.startdedicated.de) ([62.75.236.16]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Jun 2016 17:11:49 +0200 Received: from localhost (fuenf.wanjo.de.local [127.0.0.1]) by zulu1878.startdedicated.de (Postfix) with ESMTP id E5BC64D82E45; Sat, 18 Jun 2016 17:11:48 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at zulu1878.startdedicated.de Received: from zulu1878.startdedicated.de ([127.0.0.1]) by localhost (fuenf.wanjo.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ze4ibh369beA; Sat, 18 Jun 2016 17:11:47 +0200 (CEST) Received: from martins-macbook-pro.fritz.box (x4e301e41.dyn.telefonica.de [78.48.30.65]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by zulu1878.startdedicated.de (Postfix) with ESMTPSA id 5CD474D825FD; Sat, 18 Jun 2016 17:11:47 +0200 (CEST) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: multipart/signed; boundary="Apple-Mail=_BEC194C6-1431-4666-B96F-AB64E9AD681D"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Pgp-Agent: GPGMail 2.6b2 From: =?utf-8?Q?=22Martin_R=2E_Neuh=C3=A4u=C3=9Fer=22?= In-Reply-To: <3673db34-9350-a9b1-fcd5-e7593ba0fd01@etorok.net> Date: Sat, 18 Jun 2016 17:11:42 +0200 Cc: caml-list@inria.fr Message-Id: <90E10E72-C397-47B7-A6B8-43B0E8A29D5C@marneu.com> References: <7182303B-EBFF-4CFC-A7DF-B5C7A059509F@marneu.com> <3673db34-9350-a9b1-fcd5-e7593ba0fd01@etorok.net> To: =?utf-8?Q?T=C3=B6r=C3=B6k_Edwin?= X-Mailer: Apple Mail (2.3124) Subject: Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module... --Apple-Mail=_BEC194C6-1431-4666-B96F-AB64E9AD681D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 >=20 > Am 17.06.2016 um 21:41 schrieb T=C3=B6r=C3=B6k Edwin : >=20 > On 06/17/2016 09:09 PM, "Martin R. Neuh=C3=A4u=C3=9Fer" wrote: >> Dear all, >>=20 >> after an intense week of debugging some large C-bindings, I presume to h= ave found a bug in OCaml=E2=80=99s garbage collection code or in its Weak m= odule. >> To summarize, it seems as if OCaml=E2=80=99s garbage collector and its W= eak module sometimes release a custom block (by calling its finalizer) too = early, i.e. when it is still reachable. >>=20 >> I hesitate a bit before opening an =E2=80=9Eofficial=E2=80=9C issue on M= antis as I might very well overlook some detail. Therefore I=E2=80=99d like= to double-check some of the assumptions that I have made when writing my C= -stubs. Any corrections are highly welcome: >> 1. Custom blocks may be moved around in memory by the GC, but they are n= ever duplicated. >> 2. The finalizer for each custom block that is allocated by caml_alloc_c= ustom is called at most once. >> 3. The finalizer is never called for blocks that are still alive; stated= otherwise, a block that has been finalized can never been presented as a v= alue to a C-stub anymore. >=20 > Shallow copies don't seem very safe in the presence of out-of-OCaml-heap = C pointers [*]. > Perhaps Weak.get_copy should raise if it encounters a Custom_tag? Just forbidding to use custom blocks in weak sets is harsh, isn=E2=80=99t i= t? For example, we are relying on SMT solvers and use weak sets on the OCam= l side to do a lightweight hash-consing (actually, we only want to avoid ha= ving duplicate term representations in OCaml). Moreover, I suspect it might be difficult to detect situations that involve= custom blocks, as a custom block may occur deeply nested in a structured b= lock that is stored in a weak set... >=20 > The custom value may contain C pointers, that may get invalidated or chan= ged if the custom value has a finalizer (as in the testcase): > When the original weak value has no more references it gets the finalizer= invoked, and gets garbage collected > (in this case it changes unique_id to c_data_invalid_id, but it might as = well free it instead causing a crash). >=20 > Meanwhile the copy made through Weak.get_copy still lives, and shares the= copied custom value (pointer) from the weak value that got finalized. > Operations on the copied value (e.g. comparison/hashing) will try to acce= ss the custom value, which points to c_data_invalid_id. I uploaded a second example `gcbug2.ml` which crashes as well but this time= , it is designed not to check the =E2=80=9Evalidity=E2=80=9C of custom bloc= ks while a Weak.find operation is ongoing. Actually, in my experiments, thi= s version aborts within the C-layer finalizer when trying to finalize a cus= tom block that has been finalized before. So something like a double-free i= s happening here, and outside the scope of Weak.find. If the behavior is related to the shallow copies created by Weak.get_copy, = is it the case that they are finalized as well, and independently of their = source custom block? >=20 >=20 >> There is a small example program available on github: https://github.com= /martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violat= ed. >=20 > [*] AFAICT a shallow copy is created by WeakDummySet.find -> Weak.get_copy >=20 > Best regards, > -- > Edwin T=C3=B6r=C3=B6k | Co-founder and Lead Developer >=20 > Skylable open-source object storage: reliable, fast, secure > http://www.skylable.com >=20 > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs --Apple-Mail=_BEC194C6-1431-4666-B96F-AB64E9AD681D Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAldlZLIACgkQbEbxnBb9spgKvgCaAqMeLUZX1UHeFNuo7s0ZllqJ HLUAn0YNJGeRLmvI7uvDu8Lv5wk011y4 =/AUd -----END PGP SIGNATURE----- --Apple-Mail=_BEC194C6-1431-4666-B96F-AB64E9AD681D--