From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id AB45D7EE25 for ; Wed, 6 Nov 2013 14:10:17 +0100 (CET) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=pra; client-ip=212.227.126.171; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=mailfrom; client-ip=212.227.126.171; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of postmaster@moutng.kundenserver.de designates 212.227.126.171 as permitted sender) identity=helo; client-ip=212.227.126.171; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="postmaster@moutng.kundenserver.de"; x-conformance=sidf_compatible; x-record-type="v=spf1" X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Ah8EAMQ+elLU436rlGdsb2JhbABagz+DW7w1gSMWDgEBAQEHCwsJEiqCJQEBBAEjMiQQC0ICAlcGEwmHcgoJrBmSRYkyhgEmB4JrgUUDjwKKOYUOjnQ X-IPAS-Result: Ah8EAMQ+elLU436rlGdsb2JhbABagz+DW7w1gSMWDgEBAQEHCwsJEiqCJQEBBAEjMiQQC0ICAlcGEwmHcgoJrBmSRYkyhgEmB4JrgUUDjwKKOYUOjnQ X-IronPort-AV: E=Sophos;i="4.93,647,1378850400"; d="asc'?scan'208";a="34018046" Received: from moutng.kundenserver.de ([212.227.126.171]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/RC4-SHA; 06 Nov 2013 14:10:17 +0100 Received: from office1.lan.sumadev.de (dslb-088-069-130-251.pools.arcor-ip.net [88.69.130.251]) by mrelayeu.kundenserver.de (node=mreu4) with ESMTP (Nemesis) id 0Lirv8-1WBiOn21Es-00dCxW; Wed, 06 Nov 2013 14:10:15 +0100 Received: from [192.168.0.110] (ip-109-90-191-98.unitymediagroup.de [109.90.191.98]) by office1.lan.sumadev.de (Postfix) with ESMTPSA id 14D32C00D3; Wed, 6 Nov 2013 14:10:15 +0100 (CET) Message-ID: <1383743414.4083.47.camel@zotac> From: Gerd Stolpmann To: Jean Krivine Cc: caml users Date: Wed, 06 Nov 2013 14:10:14 +0100 In-Reply-To: References: Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-Z8hnYr/UWLSmPFokD/07" X-Mailer: Evolution 3.2.3-0ubuntu6 Mime-Version: 1.0 X-Provags-ID: V02:K0:LellHvlw6fl3JW2XX5lZZ79GzvS+CVhqga01vKYykWc DiwJijKz6joEcuU7K4eTXvhHgaHSdgjIaHEhyvKM8JAfe1SJZi 0KLsNpaOckV/lCTZKMv3Qic7SzXXSrW0wqZhMP/cl4prEWJT56 wCf3cRNyI2OnffyQkpPS2Nv756CV4mBFzGkMYDPE4+/3AQWDA6 FcT2HH2kpiwgUvH9SY53X8ntI6CREFOx+zW+J2sQTtG8TQl/0h /sYCh+oG73reO0XJvIb5Pe2jPvRhXa3k73AL1OooxN9i2iN559 9y3B9T7TgN0u8NUVBfNzuq4Ekbfk/6YPXAiwkxbIAVtHBoQcVH LUS1Y/ui2a60L9Lr2PLfepU+xuJxg9J6862BCskDi Subject: Re: [Caml-list] out-of-the-heap 'a arrays ? --=-Z8hnYr/UWLSmPFokD/07 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Am Dienstag, den 05.11.2013, 18:07 +0100 schrieb Jean Krivine: > Dear all >=20 >=20 > I am developing a graph rewriting algorithm which operates on large > graphs. Because of the large data structure the GC becomes quite > inefficient for two reasons that I am inferring:=20 > 1/ there is no correlation between the time of allocation of an object > and its likelihood to be garbage collected. > 2/ even when there is nothing to collect, I guess that the GC is still > inspecting the heap. >=20 >=20 > Point 1 is inducing some memory leak and point 2 is just inefficient. > I think I took care of point 1 by using my own allocation heap (so > there is nothing to collect for the GC). But to take care of point 2 I > guess I need to tell the GC that my heap (an extensible array) should > not be inspected. >=20 >=20 > As far as I understand there is a module Ancient which I can use to > tell the GC to ignore my array but, if I understand well, it would > only work if I use my array in a read only fashion.=20 > I also thought I could use Bigarray, but it seems it can only be used > for basic array types. >=20 >=20 > To summarize my question: is there a (reasonable) way to implement an > 'a array out of the ocaml heap ?=20 Yes, but it's cumbersome. I did that for the Netmulticore library of Ocamlnet. Here are the basics: You can have a pointer from the normal heap to other memory, and the GC will not follow it. You cannot have pointers the other way round, because the GC may move in-heap memory, and there is no mechanism to update such inverse pointers. In Ocamlnet you find the required support functions in http://projects.camlcity.org/projects/dl/ocamlnet-3.7.3/doc/html-main/Netsy= s_mem.html. This functionality shares the same basic ideas of Ancient, but = is more complete, and especially supports read-write modifications of out-o= f-heap values in a reasonable way. Out-of-heap memory is here encapsulated = as bigarrays. With Netsys_mem.init_array you can initialize bigarrays so th= eir contents can be interpreted as Ocaml array. With Netsys_mem.init_value = you can copy arbitrary values to bigarrays (i.e. for initializing/setting t= he elements of the array). Netsys_mem.as_value returns the pointer to the s= tructure in the bigarray as "normal" OCaml value pointer. E.g. type elem =3D { n : int } type arr =3D elem array let mem_size =3D 100000 let arr_size =3D 10 let mem =3D Bigarray.Array1.create Bigarray.char Bigarray.c_layout mem_size let (offs,blen) =3D Netsys_mem.init_array mem 0 arr_size let arr_ooh =3D Netsys_mem.as_value mem offs Now arr_ooh contains invalid pointers (which doesn't matter for the moment because the GC will not inspect them). Here is how to set all elements to some contents: let next =3D ref blen for k =3D 0 to arr_size-1 do let v =3D { n =3D 5*k } in (* some random contents *) let (v_offs, v_blen) =3D Netsys_mem.init_value mem !next v [] in let v_ooh =3D Netsys_mem.as_value mem v_offs in arr_ooh.(k) <- v_ooh; (* out-of-heap assignment, see below *) next :=3D !next + v_blen done Of course, you need to do your own memory-management here (there are higher-level functions in Ocamlnet for that, see the Netmulticore library). So finally you get an initialized out-of-heap array arr_ooh residing within the bigarray. The assignment arr_ooh.(k) <- v_ooh needs some further discussion. Until OCaml-4.00.1 this was fully supported by the OCaml runtime. OCaml-4.01.0 includes a change that disallows to modify out-of-heap memory with normal OCaml assignment operators. Ocamlnet contains a workaround (which works by overriding the changed caml_initialize and caml_modify functions with their old definitions), and it is automatically enabled if you add -package netmulticore at link time. The workaround is incompatible with non-custom bytecode links, though. Gerd >=20 >=20 > Thanks! > JK --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ --=-Z8hnYr/UWLSmPFokD/07 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAABAgAGBQJSej+2AAoJEAaM4b9ZLB5TN3QIAJioVWjl0D3Tns+C9xRe9Gwy G1Nhk1WY+mrF73XUK5Umc7duPL0jCgHl0ylt0MFNohHXJVrwoRXSgdGr4a+Ltp6q BhZa6m6sCXMGA7vL50rV74EWHZ8cg8eGQayyBaBHVLUIa06EHRwaOb9M1UVbYg8f vZ89+4Oahe4+uKI9jiOpIcGOIQGIVZbGHQPua6ess/RUv2/Dik4ZchaJlFGt/M2G b76vegCqCTofYc5BMqLe046k8Vm3oGjzS8op1bmXn3SnKbS7koO3u41MO1Ep1cB2 ybpteB+fY/FAjkvFXdtGIGU0OpfIrExQq58wA+Q3IutMqTbBVan7LnsWYP5aLys= =ISyg -----END PGP SIGNATURE----- --=-Z8hnYr/UWLSmPFokD/07--