From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id 2E9857F75C for ; Sun, 10 Aug 2014 08:51:57 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=pra; client-ip=212.227.17.10; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of info@gerd-stolpmann.de) identity=mailfrom; client-ip=212.227.17.10; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="info@gerd-stolpmann.de"; x-conformance=sidf_compatible Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.kundenserver.de) identity=helo; client-ip=212.227.17.10; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="info@gerd-stolpmann.de"; x-sender="postmaster@mout.kundenserver.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Au0BAOgV51PU4xEKlWdsb2JhbABZg19XgnjKIAqHSAGBBBYQAQEBAQcNCQkSK4QEAQUjJgwkEAsONAICVwYTCYg9Ca9blGgXiWOFQyYHgjgPMhKBQQWGD4tCkgAFkTJqAQ X-IPAS-Result: Au0BAOgV51PU4xEKlWdsb2JhbABZg19XgnjKIAqHSAGBBBYQAQEBAQcNCQkSK4QEAQUjJgwkEAsONAICVwYTCYg9Ca9blGgXiWOFQyYHgjgPMhKBQQWGD4tCkgAFkTJqAQ X-IronPort-AV: E=Sophos;i="5.01,835,1400018400"; d="asc'?scan'208";a="74422723" Received: from mout.kundenserver.de ([212.227.17.10]) by mail3-smtp-sop.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 10 Aug 2014 08:51:56 +0200 Received: from office1.lan.sumadev.de (dslb-088-068-023-111.088.068.pools.vodafone-ip.de [88.68.23.111]) by mrelayeu.kundenserver.de (node=mreue105) with ESMTP (Nemesis) id 0LbacF-1WZvUm1ZoB-00lGVX; Sun, 10 Aug 2014 08:51:55 +0200 Received: from [192.168.89.102] (ip-109-45-1-113.web.vodafone.de [109.45.1.113]) by office1.lan.sumadev.de (Postfix) with ESMTPSA id E4A62DC270; Sun, 10 Aug 2014 08:51:53 +0200 (CEST) Message-ID: <1407653503.5797.28.camel@e130> From: Gerd Stolpmann To: Yoann Padioleau Cc: Caml List Date: Sun, 10 Aug 2014 08:51:43 +0200 In-Reply-To: <7F92DE4E-875E-4020-AF4F-5BC19080225A@fb.com> References: <7F92DE4E-875E-4020-AF4F-5BC19080225A@fb.com> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-vxMaP81lGSgLvQ5E0wMJ" X-Mailer: Evolution 3.10.4-0ubuntu1 Mime-Version: 1.0 X-Provags-ID: V02:K0:XKwPfgxRfDDmMvoOlnnnvZZ3scFJWeBW+87Qh7PB2X5 kYh9ot3rUutT8laZAne9mz1ZJFeUvVLpE5Gvil5X2ueMxGifLx nIarHXk1heyrsF3pPgIKadDDybg/mMa9TfNdsVZzvZ3iRfQDwo wPGdphuurjH04sDyHWdaXPQK7M9cOgXNjDBrjPRK5jTlqv79DX 9/F2RZJW+p6PllcXqZBfjuXpOaEWm10y+umFBH8SbLml6+49iq Z0VQNrYm+EuLk3o0WaEETZ+OK/niYEeTZw9dni/0GZAY0mUbD9 NFETmoTwzBPYUe8d0JVPAzebXXdDoask5bFgIFAkALdWPeAOb4 xrd/GCWpJhvYdfECoOZ0aVu1m/S9ADmBoNYKYVF7x X-UI-Out-Filterresults: notjunk:1; Subject: Re: [Caml-list] strategies to deal with huge in-memory "object" graphs? --=-vxMaP81lGSgLvQ5E0wMJ Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Am Freitag, den 08.08.2014, 22:20 +0000 schrieb Yoann Padioleau: > Hi list, >=20 > I have an application that is gradually creating a graph (using ocamlgrap= h) and=20 > the amount of memory it is using is around 3 or 4 Gb (my machine has=20 > 74Gb of RAM). There are lots of nodes and edges in this graph. The proble= m is that building > this graph takes a huge amount of time. As the build progresses, it gets = slower > and slower. My guess is that the =E2=80=9Cobject=E2=80=9D graph is gettin= g really huge and > so the Gc needs to explore each time even more. I=E2=80=99ve tried things= like >=20 > (* see www.elehack.net/michael/blog/2010/06/ocaml-memory-tuning *) > Gc.set { (Gc.get()) with Gc.minor_heap_size =3D 4_000_000 }; > (* goes from 5300s to 3000s for building db for www *) > Gc.set { (Gc.get()) with Gc.major_heap_increment =3D 8_000_000 }; > Gc.set { (Gc.get()) with Gc.space_overhead =3D 300 }; >=20 >=20 > but it does not really help. It is still really slow. >=20 > In the past I sometimes use the Marshall module to reduce the number of = =E2=80=9Cobjects=E2=80=9D, > but it forces me to rewrite quite a lot the code.=20 >=20 > Is there a way to partition the heap so that for instance in my case all = the graph > related things are put in a different area that the Gc does not have to e= xplore each time. > I=E2=80=99d like a minor heap, major heap, and then a do_not_gc_this_hea= p_it_is_only_growing_there_is_no_garbage_here_to_collect. The latter can be accomplished by setting space_overhead to a large value (maybe 1E6). Also set max_overhead to 1E6 to avoid compactions. Once you have built the graph, you can move the whole beast (provided it is read-only) to a non-GC-managed area with either Ancient (simpler to use, fewer features), or Ocamlnet's Netmulticore. But you really can only move the whole graph, with everything that is reachable from it. Any mutation will kill the program. Gerd >=20 >=20 --=20 ------------------------------------------------------------ Gerd Stolpmann, Darmstadt, Germany gerd@gerd-stolpmann.de My OCaml site: http://www.camlcity.org Contact details: http://www.camlcity.org/contact.html Company homepage: http://www.gerd-stolpmann.de ------------------------------------------------------------ --=-vxMaP81lGSgLvQ5E0wMJ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQEcBAABAgAGBQJT5xZ/AAoJEAaM4b9ZLB5TQmoH/RPaGHCR9ftuFlETbtfQneq8 H1ntl4MLEYw+llzRz+UHil9PjqHAes5v2bGEAuM8Vmm6trc8p0Xe297hJHo/ipn4 FntqJDf5Yhp42eW7nDrOazKqAUbJtZX+LxSJPC2m2/DrpeGzEopI3BBFIy2FZh9Z 1gMHNumMmXYNPXXlvFNEGiOrAbwpBn5A01+GSiwjJG0xVCBFAWTnGQ71M9lgYGc7 kM8TC296NXHl57iMAwakILGsyXP+1h5+eUa8MKfbUuPRCwO4GjkiHjMqfRJIAgUV Qu6zv/L+PSa0O0UWDbrzuWECleF/8Rn/CerqhrabYMseeTc2/dYDLr+luBvefSU= =gSz1 -----END PGP SIGNATURE----- --=-vxMaP81lGSgLvQ5E0wMJ--