From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by sympa.inria.fr (Postfix) with ESMTPS id DB6BC7F747 for ; Sat, 9 Aug 2014 00:21:05 +0200 (CEST) Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of pad@fb.com) identity=pra; client-ip=67.231.145.42; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="prvs=3297f5d1c5=pad@fb.com"; x-sender="pad@fb.com"; x-conformance=sidf_compatible Received-SPF: Pass (mail3-smtp-sop.national.inria.fr: domain of prvs=3297f5d1c5=pad@fb.com designates 67.231.145.42 as permitted sender) identity=mailfrom; client-ip=67.231.145.42; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="prvs=3297f5d1c5=pad@fb.com"; x-sender="prvs=3297f5d1c5=pad@fb.com"; x-conformance=sidf_compatible; x-record-type="v=spf1" Received-SPF: None (mail3-smtp-sop.national.inria.fr: no sender authenticity information available from domain of postmaster@mx0a-00082601.pphosted.com) identity=helo; client-ip=67.231.145.42; receiver=mail3-smtp-sop.national.inria.fr; envelope-from="prvs=3297f5d1c5=pad@fb.com"; x-sender="postmaster@mx0a-00082601.pphosted.com"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AmICAEVM5VND55Eqf2dsb2JhbABag19XBK4LAQEBBp5OCohZFhABAQkNCggUKYQKQAEBBzERAYEAJwSIVQ2vDwEBhXgBBZB5BoV8jA8PRCSBHLR9bIFH X-IPAS-Result: AmICAEVM5VND55Eqf2dsb2JhbABag19XBK4LAQEBBp5OCohZFhABAQkNCggUKYQKQAEBBzERAYEAJwSIVQ2vDwEBhXgBBZB5BoV8jA8PRCSBHLR9bIFH X-IronPort-AV: E=Sophos;i="5.01,827,1400018400"; d="scan'208";a="74351104" Received: from mx0a-00082601.pphosted.com ([67.231.145.42]) by mail3-smtp-sop.national.inria.fr with ESMTP; 09 Aug 2014 00:21:04 +0200 Received: from pps.filterd (m0044008 [127.0.0.1]) by mx0a-00082601.pphosted.com (8.14.5/8.14.5) with SMTP id s78MKRs3003471 for ; Fri, 8 Aug 2014 15:21:01 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=fb.com; h=from : to : subject : date : message-id : content-type : content-id : content-transfer-encoding : mime-version; s=facebook; bh=Cmz/YnH8dDnfgGEvLx15GGQd74TxEet7mKy0Muf5+8Y=; b=JaycG2RvuxxxsWoyw757Dz/2sOPOFMI7MchpKPezCRZVZKsdGx/nMzEtlvYzMWMQvX6M l4JwqYNsMdvQANg1rcWhrUkVKj41LbLjboA0C/Sy/GI5O2bx1zEBBC2vifiKHJJHQKxB C6OCdWsezQ7uyvv+G84HkshzCZ02cDVdlKo= Received: from mail.thefacebook.com (mailwest.thefacebook.com [173.252.71.148]) by mx0a-00082601.pphosted.com with ESMTP id 1nmu8p9cgs-1 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=OK) for ; Fri, 08 Aug 2014 15:21:01 -0700 Received: from PRN-MBX02-4.TheFacebook.com ([169.254.5.208]) by PRN-CHUB04.TheFacebook.com ([fe80::7ded:c10e:ef04:80d8%12]) with mapi id 14.03.0195.001; Fri, 8 Aug 2014 15:20:59 -0700 From: Yoann Padioleau To: Caml List Thread-Topic: strategies to deal with huge in-memory "object" graphs? Thread-Index: AQHPs1cH3o81xdIiMk+9drQvxs5AyQ== Date: Fri, 8 Aug 2014 22:20:59 +0000 Message-ID: <7F92DE4E-875E-4020-AF4F-5BC19080225A@fb.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.16.4] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.12.52,1.0.27,0.0.0000 definitions=2014-08-08_05:2014-08-08,2014-08-08,1970-01-01 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 kscore.is_bulkscore=2.08916506150292e-07 kscore.compositescore=0 circleOfTrustscore=0.832 compositescore=0.999802048357014 urlsuspect_oldscore=0.999802048357014 suspectscore=0 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=0 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=104 rbsscore=0.999802048357014 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1408080268 X-FB-Internal: deliver Subject: [Caml-list] strategies to deal with huge in-memory "object" graphs? Hi list, I have an application that is gradually creating a graph (using ocamlgraph)= and=20 the amount of memory it is using is around 3 or 4 Gb (my machine has=20 74Gb of RAM). There are lots of nodes and edges in this graph. The problem = is that building this graph takes a huge amount of time. As the build progresses, it gets sl= ower and slower. My guess is that the =93object=94 graph is getting really huge = and so the Gc needs to explore each time even more. I=92ve tried things like (* see www.elehack.net/michael/blog/2010/06/ocaml-memory-tuning *) Gc.set { (Gc.get()) with Gc.minor_heap_size =3D 4_000_000 }; (* goes from 5300s to 3000s for building db for www *) Gc.set { (Gc.get()) with Gc.major_heap_increment =3D 8_000_000 }; Gc.set { (Gc.get()) with Gc.space_overhead =3D 300 }; but it does not really help. It is still really slow. In the past I sometimes use the Marshall module to reduce the number of =93= objects=94, but it forces me to rewrite quite a lot the code.=20 Is there a way to partition the heap so that for instance in my case all th= e graph related things are put in a different area that the Gc does not have to exp= lore each time. I=92d like a minor heap, major heap, and then a do_not_gc_this_heap_it_is_= only_growing_there_is_no_garbage_here_to_collect.