From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail1-relais-roc.national.inria.fr (mail1-relais-roc.national.inria.fr [192.134.164.82]) by yquem.inria.fr (Postfix) with ESMTP id 7EACEBC57 for ; Thu, 16 Dec 2010 13:39:18 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqMAADOXCU3RVdg0kGdsb2JhbACDZKBJCBUBAQEBCQkMBxEEIKZ2iTQ8ghiFMC6IVwEBAwWBHIM1dASKfYYK X-IronPort-AV: E=Sophos;i="4.59,354,1288566000"; d="scan'208";a="92507196" Received: from mail-qw0-f52.google.com ([209.85.216.52]) by mail1-smtp-roc.national.inria.fr with ESMTP; 16 Dec 2010 13:39:17 +0100 Received: by qwi4 with SMTP id 4so3055058qwi.39 for ; Thu, 16 Dec 2010 04:39:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:cc:content-type; bh=IN07cGhaJRD0z2uLvOaro+0+UzzyiRaDl3+nBAunHbk=; b=pjR/iMdlQFKOztDTAuCc+CwRE/rJp+2aVoKPQIwWHSTzpwgyByYtQXldXVT8hL7hzt yQu2XIMkhLLVIWYV1kECOg+bKbEIeH/FtE/quuaIAkayG41QsVBnhN8/F+klu6E5G+ds BhCLIdGvmJqwvpoTVUfyF3vdSbQKMLfS5u7hY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=Dx7FmpP3di5i+QKjYHpJfg/Q0qJ5YyEKrRH9K/qaWKiR7+h2kkp8+395gxCn1hwsAB ji0nuWbMvjDL+q8Z31G6glfArgBTFGJTJNsA5qnGSjNljnIZnt/TKzLyXEGqseKwdVR1 wAU6IoMTa1RrXE13v0OwlPcYxaDCUKnwSLmEI= Received: by 10.229.185.209 with SMTP id cp17mr1632543qcb.159.1292503157295; Thu, 16 Dec 2010 04:39:17 -0800 (PST) MIME-Version: 1.0 Received: by 10.229.233.9 with HTTP; Thu, 16 Dec 2010 04:38:57 -0800 (PST) In-Reply-To: <8D1E9D5B-188C-4AF3-980B-EF229BA98FB4@gmail.com> References: <8D1E9D5B-188C-4AF3-980B-EF229BA98FB4@gmail.com> From: Jesper Louis Andersen Date: Thu, 16 Dec 2010 13:38:57 +0100 Message-ID: Subject: Re: [Caml-list] appending data to a mmap-ed file To: Joel Reymont Cc: caml-list@yquem.inria.fr Content-Type: text/plain; charset=UTF-8 X-Spam: no; 0.00; appending:01 appending:01 erlang:01 erlang:01 ocaml:01 ocaml:01 read-only:01 andersen:98 andersen:98 wrote:01 ints:01 caml-list:01 strings:01 behaviour:01 data:02 On Thu, Dec 16, 2010 at 12:31, Joel Reymont wrote: > I'm constantly appending to a file of stock quotes (ints, longs, doubles, etc.). I have this file mapped into memory with mmap. Ok, this helps a bit on what you are trying to do (you asked almost the same question on the Erlang mailing list, but the details of getting a foothold for the same thing in Erlang is subtly different) My approach would be simple by noting you you two kinds of data and some peculiar behaviour * "Newly generated data" * "Old data for archeology" * Data are almost never deleted So: * If data is less than a threshold in size (preferably less than a couple of PAGE_SIZE page boundaries, keep data in memory and serve it from there. Simply have an Ocaml array of bytes or something such to store data into (my Ocaml representation specific knowledge is not up to par at the moment, but arrange it such that the byte-array has C-representation underneath. I know that Ocaml strings have this). This is the newly generated data. * Once in a while, you write(2) this string to the file on the disk, then reopen the mmap() (which is now READ-ONLY as an effect. There might be sharing tricks to play here should you do multi-process). * Lookup is handled by checking if data is archeology or data are recent. The right lookup is then made. Everything hidden by batching it up in a module. * You can play with the factor of when to write data to disk. Too large, and you risk loosing too much data on failure. Too small and the approach dies of syscall-overhead. You may have additional constraints, so spill them, please. -- J.