From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id A12737FECC for ; Fri, 17 Jun 2016 22:29:44 +0200 (CEST) IronPort-PHdr: 9a23:0ru2IREP19vkKtPqoke0fZ1GYnF86YWxBRYc798ds5kLTJ74p8mwAkXT6L1XgUPTWs2DsrQf27uQ4virADBeqb+681k8M7V0HycfjssXmwFySOWkMmbcaMDQUiohAc5ZX0Vk9XzoeWJcGcL5ekGA6ibqtW1aJBzzOEJPK/jvHcaK1oLsh7H0qsWYOlkWzBOGIppMbzyO5T3LsccXhYYwYo0Q8TDu5kVyRuJN2GlzLkiSlRuvru25/Zpk7jgC86l5r50IeezAcq85Vb1VCig9eyBwvZWz9Er1dhaU/nYXTkkRlxNJBUCFsEC7Dd/NtX7GKPDj2x63MdL3S/hgVDi97qEoTRutjywOMjoR/X3QkMtzyqRWvETyiQZ4xtueRYiPKPt4SZmbSvUXWGrNQ3FcTWYJVpu7dJcnCuMbO+tF6YL6og1d/lOFGQCwCba3mXdzjXjs0Ph/jr0s Authentication-Results: mail3-smtp-sop.national.inria.fr; spf=None smtp.pra=post@marneu.com; spf=Pass smtp.mailfrom=post@marneu.com; spf=None smtp.helo=postmaster@zulu1878.startdedicated.de Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of post@marneu.com) identity=pra; client-ip=62.75.236.16; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="post@marneu.com"; x-sender="post@marneu.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of post@marneu.com designates 62.75.236.16 as permitted sender) identity=mailfrom; client-ip=62.75.236.16; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="post@marneu.com"; x-sender="post@marneu.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@zulu1878.startdedicated.de) identity=helo; client-ip=62.75.236.16; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="post@marneu.com"; x-sender="postmaster@zulu1878.startdedicated.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: A0CLDAArXWRX/xDsSz5dhBRLATGCfqZJkRyBeiJghRUCgV4TAQEBAQEBAQFkJ4IxghoBAQEDAQwXBC4ZCwUJAgkCGA0dAgIbKhIGExKIBAMPDAEJkxmdHYxjA4NoAQEBAQEFAQEBAQEBARIOBYUqgm8Igk6ELASDESuCLwEEhkoMkhsBgy2CV4gkgjeHCoVhj3MCIAIyggcdF4E3bIg4gUMBAQE X-IPAS-Result: A0CLDAArXWRX/xDsSz5dhBRLATGCfqZJkRyBeiJghRUCgV4TAQEBAQEBAQFkJ4IxghoBAQEDAQwXBC4ZCwUJAgkCGA0dAgIbKhIGExKIBAMPDAEJkxmdHYxjA4NoAQEBAQEFAQEBAQEBARIOBYUqgm8Igk6ELASDESuCLwEEhkoMkhsBgy2CV4gkgjeHCoVhj3MCIAIyggcdF4E3bIg4gUMBAQE X-IronPort-AV: E=Sophos;i="5.26,485,1459807200"; d="asc'?scan'208";a="181669603" Received: from wanjo.de (HELO zulu1878.startdedicated.de) ([62.75.236.16]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Jun 2016 22:29:43 +0200 Received: from localhost (fuenf.wanjo.de.local [127.0.0.1]) by zulu1878.startdedicated.de (Postfix) with ESMTP id DDF254D837C5; Fri, 17 Jun 2016 22:29:41 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at zulu1878.startdedicated.de Received: from zulu1878.startdedicated.de ([127.0.0.1]) by localhost (fuenf.wanjo.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 61YTOcH_YXMH; Fri, 17 Jun 2016 22:29:39 +0200 (CEST) Received: from martins-macbook-pro.fritz.box (x4e328939.dyn.telefonica.de [78.50.137.57]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by zulu1878.startdedicated.de (Postfix) with ESMTPSA id 852824D83155; Fri, 17 Jun 2016 22:29:39 +0200 (CEST) Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: multipart/signed; boundary="Apple-Mail=_CE108E71-8CA2-42BC-9C93-0523A0120589"; protocol="application/pgp-signature"; micalg=pgp-sha1 X-Pgp-Agent: GPGMail 2.6b2 From: =?utf-8?Q?=22Martin_R=2E_Neuh=C3=A4u=C3=9Fer=22?= In-Reply-To: <3673db34-9350-a9b1-fcd5-e7593ba0fd01@etorok.net> Date: Fri, 17 Jun 2016 22:29:35 +0200 Cc: caml-list@inria.fr Message-Id: <01CFEDAE-C1CB-4968-9FDF-1EC004CC027A@marneu.com> References: <7182303B-EBFF-4CFC-A7DF-B5C7A059509F@marneu.com> <3673db34-9350-a9b1-fcd5-e7593ba0fd01@etorok.net> To: =?utf-8?Q?T=C3=B6r=C3=B6k_Edwin?= X-Mailer: Apple Mail (2.3124) Subject: Re: [Caml-list] Presumed bug in OCaml's garbage collector or in its Weak module... --Apple-Mail=_CE108E71-8CA2-42BC-9C93-0523A0120589 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Thanks for the good point! And yes, the weak sets use Weak.get_copy interna= lly. However, the exact semantics of Weak.get_copy is still a bit unclear to me.= It creates a =E2=80=9Eshallow copy=E2=80=9C of a value that might independ= ently be garbage collected. If that shallow copy can survive the finalizati= on of its source, that might cause the problem. Are there any restrictions = enforced on values obtained by Weak.get_copy? Or could this even be misused= to create arbitrary copies of custom blocks that propagate anywhere? I will try to extend the example such that maybe it triggers the problem ou= tside the compare and hash functions, i.e. not in one of the functions that= the weak set code might expose to a shallow copy. Nevertheless, the behavior seems to have changes somewhere between OCaml 4.= 01.1 and 4.02.0. And the Mantis tickets 7161 and 7157 seem to affect OCaml= =E2=80=99s behavior, as well. That might explain that the testcase succeeds= with the first beta of OCaml 4.03.0=E2=80=A6 Best, Martin > Am 17.06.2016 um 21:41 schrieb T=C3=B6r=C3=B6k Edwin : >=20 > On 06/17/2016 09:09 PM, "Martin R. Neuh=C3=A4u=C3=9Fer" wrote: >> Dear all, >>=20 >> after an intense week of debugging some large C-bindings, I presume to h= ave found a bug in OCaml=E2=80=99s garbage collection code or in its Weak m= odule. >> To summarize, it seems as if OCaml=E2=80=99s garbage collector and its W= eak module sometimes release a custom block (by calling its finalizer) too = early, i.e. when it is still reachable. >>=20 >> I hesitate a bit before opening an =E2=80=9Eofficial=E2=80=9C issue on M= antis as I might very well overlook some detail. Therefore I=E2=80=99d like= to double-check some of the assumptions that I have made when writing my C= -stubs. Any corrections are highly welcome: >> 1. Custom blocks may be moved around in memory by the GC, but they are n= ever duplicated. >> 2. The finalizer for each custom block that is allocated by caml_alloc_c= ustom is called at most once. >> 3. The finalizer is never called for blocks that are still alive; stated= otherwise, a block that has been finalized can never been presented as a v= alue to a C-stub anymore. >=20 > Shallow copies don't seem very safe in the presence of out-of-OCaml-heap = C pointers [*]. > Perhaps Weak.get_copy should raise if it encounters a Custom_tag? >=20 > The custom value may contain C pointers, that may get invalidated or chan= ged if the custom value has a finalizer (as in the testcase): > When the original weak value has no more references it gets the finalizer= invoked, and gets garbage collected > (in this case it changes unique_id to c_data_invalid_id, but it might as = well free it instead causing a crash). >=20 > Meanwhile the copy made through Weak.get_copy still lives, and shares the= copied custom value (pointer) from the weak value that got finalized. > Operations on the copied value (e.g. comparison/hashing) will try to acce= ss the custom value, which points to c_data_invalid_id. >=20 >=20 >> There is a small example program available on github: https://github.com= /martin-neuhaeusser/ocaml_bug where the above assumptions seem to be violat= ed. >=20 > [*] AFAICT a shallow copy is created by WeakDummySet.find -> Weak.get_copy >=20 > Best regards, > -- > Edwin T=C3=B6r=C3=B6k | Co-founder and Lead Developer >=20 > Skylable open-source object storage: reliable, fast, secure > http://www.skylable.com >=20 > -- > Caml-list mailing list. Subscription management and archives: > https://sympa.inria.fr/sympa/arc/caml-list > Beginner's list: http://groups.yahoo.com/group/ocaml_beginners > Bug reports: http://caml.inria.fr/bin/caml-bugs -- Martin Neuh=C3=A4u=C3=9Fer Phone: +49 (911) 49051223 Normannenstra=C3=9Fe 15 Mobile: +49 (172) 8966488 D-90461N=C3=BCrnberg GnuPG: 0x16FDB298 --Apple-Mail=_CE108E71-8CA2-42BC-9C93-0523A0120589 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iEYEARECAAYFAldkXbIACgkQbEbxnBb9sphJbACfZHk0DYTVdwI+N34SmplTysCm Js4AnR4hmIB0kLJ33zo0ZBmxOmvogMfq =g/hl -----END PGP SIGNATURE----- --Apple-Mail=_CE108E71-8CA2-42BC-9C93-0523A0120589--