From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@sympa.inria.fr Delivered-To: caml-list@sympa.inria.fr Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by sympa.inria.fr (Postfix) with ESMTPS id 44FDC7EE25 for ; Tue, 5 Nov 2013 20:52:31 +0100 (CET) Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of goswin-v-b@web.de) identity=pra; client-ip=212.227.15.14; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="goswin-v-b@web.de"; x-sender="goswin-v-b@web.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of goswin-v-b@web.de) identity=mailfrom; client-ip=212.227.15.14; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="goswin-v-b@web.de"; x-sender="goswin-v-b@web.de"; x-conformance=sidf_compatible Received-SPF: None (mail2-smtp-roc.national.inria.fr: no sender authenticity information available from domain of postmaster@mout.web.de) identity=helo; client-ip=212.227.15.14; receiver=mail2-smtp-roc.national.inria.fr; envelope-from="goswin-v-b@web.de"; x-sender="postmaster@mout.web.de"; x-conformance=sidf_compatible X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AkICAL1LeVLU4w8OnGdsb2JhbABZDr19hUGBLBYOAQEBAQEGDQkJFCiCJQEBBTo/EAsYCSUPBSghiAEBFrQ9H4ojj1kHgyCBDwOYCYYqhBuKLEA X-IPAS-Result: AkICAL1LeVLU4w8OnGdsb2JhbABZDr19hUGBLBYOAQEBAQEGDQkJFCiCJQEBBTo/EAsYCSUPBSghiAEBFrQ9H4ojj1kHgyCBDwOYCYYqhBuKLEA X-IronPort-AV: E=Sophos;i="4.93,640,1378850400"; d="scan'208";a="41092839" Received: from mout.web.de ([212.227.15.14]) by mail2-smtp-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES128-SHA; 05 Nov 2013 20:51:56 +0100 Received: from frosties.localnet ([37.49.32.119]) by smtp.web.de (mrweb103) with ESMTPSA (Nemesis) id 0MNcMo-1VXkG40PBC-007A18 for ; Tue, 05 Nov 2013 20:52:30 +0100 Received: from mrvn by frosties.localnet with local (Exim 4.80) (envelope-from ) id 1Vdmft-0004ZX-EI; Tue, 05 Nov 2013 20:52:29 +0100 Date: Tue, 5 Nov 2013 20:52:29 +0100 From: Goswin von Brederlow To: Nicolas Ojeda Bar Cc: caml-list@yquem.inria.fr Message-ID: <20131105195229.GC16998@frosties> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Provags-ID: V03:K0:1Igtktr4EMZ7GgQWUl3kvfNE2CIkNJhK53Z1FSM4DtiQ+2n8GV6 cfSRMbNtSeylSyHRCaENauyWpyoNTdIxtKro7Sjmyb+Q9xD6Sb3px66EF41XKq/3mVlU0N9 LgtYiR8ie0al7EWNPjH3VoSPJvAwQmw587w2p1XNgu3Nh882lLKGw/0swLOdaZxJvuvseLx 5zpDSThGttHYEq3EFOC1A== Subject: Re: [Caml-list] Bigarray.map_file with large files? On Tue, Nov 05, 2013 at 07:10:24PM +0000, Nicolas Ojeda Bar wrote: > Dear list, > > Is it correct that one cannot map large files (> 2GB in 32-bit systems) using Bigarray.map_file? > Is there any way around this? > > Thank you very much, > Nicolas You can map large files just fine. The Bigarray.Array1.map_file takes a "?pos:int64" argument. BUT you can only have 2GB memory total on a 32-bit system with a 1:1 (obsolete) split, 3GB with 3:1 split and 3.99GB with a 32-bit thread on a 64-bit kernel. So you can only map a 1.9G/2.9G/3.9G chunk (or multiple smaller ones) at a time. So this doesn't help you that much I bet. You have to map and unmap the file in chunks and juggle them manually. There simply is not enough address space available in 32-bit. You could write (or google for) a wrapper class around Bigarray that mmaps/munmaps chunks of files on demand. You would have to use an Int64 as index variable. MfG Goswin