From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: ** X-Spam-Status: No, score=2.2 required=5.0 tests=HTML_MESSAGE,INFO_TLD, SPF_NEUTRAL autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from discorde.inria.fr (discorde.inria.fr [192.93.2.38]) by yquem.inria.fr (Postfix) with ESMTP id B1B9ABC0A for ; Wed, 3 Jan 2007 03:49:21 +0100 (CET) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.190]) by discorde.inria.fr (8.13.6/8.13.6) with ESMTP id l032nLHH015311 for ; Wed, 3 Jan 2007 03:49:21 +0100 Received: by nf-out-0910.google.com with SMTP id q29so7665894nfc for ; Tue, 02 Jan 2007 18:49:21 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=WOxQ0uDu2zn2q39VuHf1l4Fn3YgV/TaSofIlw5eZ+EMG4Z6Fu5anN3jU4whNXiUPWZX0jtxjYFsg6SWG8CnLi/XrgZyFTPpFA19AlRi+dtrRe6bNizX+b5uKijWAmARGoYQTUoaPOlJybJfrgTjQVJvHtOGA+61fM2YkfGoYjJA= Received: by 10.78.201.15 with SMTP id y15mr2101481huf.1167792560874; Tue, 02 Jan 2007 18:49:20 -0800 (PST) Received: by 10.78.16.5 with HTTP; Tue, 2 Jan 2007 18:49:20 -0800 (PST) Message-ID: Date: Tue, 2 Jan 2007 21:49:20 -0500 From: "Markus Mottl" To: "Gerd Stolpmann" Subject: Re: [Caml-list] ANNOUNCE : libsndfile-ocaml alpha Cc: "Erik de Castro Lopo" , caml-list@yquem.inria.fr In-Reply-To: <1167781575.8359.21.camel@localhost.localdomain> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_59657_11602371.1167792560804" References: <20061222155407.ebe6c272.mle+ocaml@mega-nerd.com> <20061231110831.GA21198@furbychan.cocan.org> <20070101095114.911ca90d.mle+ocaml@mega-nerd.com> <4599205A.7050802@laposte.net> <20070102065845.c6558b18.mle+ocaml@mega-nerd.com> <1167699073.13679.15.camel@rosella.wigram> <20070102121424.07da295b.mle+ocaml@mega-nerd.com> <1167765505.5341.12.camel@rosella.wigram> <20070103093701.3b827e81.mle+ocaml@mega-nerd.com> <1167781575.8359.21.camel@localhost.localdomain> X-j-chkmail-Score: MSGID : 459B19B1.000 on discorde : j-chkmail score : X : 0/20 1 0.000 -> 1 X-Miltered: at discorde with ID 459B19B1.000 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; markus:01 mottl:01 markus:01 mottl:01 gerd:01 stolpmann:01 buffer:01 buffer:01 alignment:01 bigarray:01 mappings:01 ocaml:01 bug:01 bigarrays:01 bug:01 ------=_Part_59657_11602371.1167792560804 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline On 1/2/07, Gerd Stolpmann wrote: > > Just a quick and somehow interesting observation: Linus assumes that the > read buffer is page-aligned in this comparison, something you cannot > expect in most higher languages (of course you can arrange that in > libsndfile - did you?). Right, this is an unwarranted assumption. Furthermore, for certain applications with complex access patterns I'd expect it to be way easier for the user to use mmap efficiently. Even assuming that "read" were always faster, implementing your own complex buffer management to cache data efficiently for particular access patterns may not be easy. Unless your name is Linus, of course... ;-) That reminds me that the OS guys live in another > world. Would be interesting which function wins if you compare Unix.read > (which does not care about alignment, and does an extra copy) with > Bigarray.mmap. > I use mmap in a fileserver for performance reasons. Compared to using I/O-channels it requires, if I remember correctly, only about 50% of the CPU-time, but OCaml-channels have to do an extra copy from the channel buffer to the user buffer so this is not quite a fair comparison. I'd still expect mmap to reduce overall CPU-time over "read" for larger files due to less data copying between kernel buffers and user space. I have observed that mmap does not pay for small files (my cutoff point is 8192 bytes), possibly due to some setup overhead for memory mappings. Total time is not strongly effected, because we are generally I/O-bound, but if you run the fileserver on the same machine as applications, the lower CPU-time is a noticable advantage. There is one caveat though regarding mmap with OCaml on 32bit platforms: the GC has a bug which prevents it from reclaiming bigarrays aggressively enough. Though there is plenty of RAM (the kernel need not keep mapped files in memory), the process might run out of address space. I hope this bug (0004108) will be fixed in the next release. Regards, Markus -- Markus Mottl http://www.ocaml.info markus.mottl@gmail.com ------=_Part_59657_11602371.1167792560804 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline On 1/2/07, Gerd Stolpmann <info@gerd-stolpmann.de> wrote:
Just a quick and somehow interesting observation: Linus assumes that the
read buffer is page-aligned in this comparison, something you cannot
expect in most higher languages (of course you can arrange that in
libsndfile - did you?).

Right, this is an unwarranted assumption.  Furthermore, for certain applications with complex access patterns I'd expect it to be way easier for the user to use mmap efficiently.  Even assuming that "read" were always faster, implementing your own complex buffer management to cache data efficiently for particular access patterns may not be easy.  Unless your name is Linus, of course... ;-)

That reminds me that the OS guys live in another
world. Would be interesting which function wins if you compare Unix.read
(which does not care about alignment, and does an extra copy) with
Bigarray.mmap.

I use mmap in a fileserver for performance reasons.  Compared to using I/O-channels it requires, if I remember correctly, only about 50% of the CPU-time, but OCaml-channels have to do an extra copy from the channel buffer to the user buffer so this is not quite a fair comparison.  I'd still expect mmap to reduce overall CPU-time over "read" for larger files due to less data copying between kernel buffers and user space.

I have observed that mmap does not pay for small files (my cutoff point is 8192 bytes), possibly due to some setup overhead for memory mappings.  Total time is not strongly effected, because we are generally I/O-bound, but if you run the fileserver on the same machine as applications, the lower CPU-time is a noticable advantage.

There is one caveat though regarding mmap with OCaml on 32bit platforms: the GC has a bug which prevents it from reclaiming bigarrays aggressively enough.  Though there is plenty of RAM (the kernel need not keep mapped files in memory), the process might run out of address space.  I hope this bug (0004108) will be fixed in the next release.

Regards,
Markus

--
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com ------=_Part_59657_11602371.1167792560804--