From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from mail3-relais-sop.national.inria.fr (mail3-relais-sop.national.inria.fr [192.134.164.104]) by yquem.inria.fr (Postfix) with ESMTP id 2ACFBBC37 for ; Wed, 28 Oct 2009 16:00:48 +0100 (CET) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AgcDAMr550rZSMDdi2dsb2JhbACBT5lwAQEBCgsKBxEFviKEPwSBYQ X-IronPort-AV: E=Sophos;i="4.44,640,1249250400"; d="scan'208";a="37146091" Received: from fmmailgate01.web.de ([217.72.192.221]) by mail3-smtp-sop.national.inria.fr with ESMTP; 28 Oct 2009 16:00:47 +0100 Received: from smtp05.web.de (fmsmtp05.dlan.cinetic.de [172.20.4.166]) by fmmailgate01.web.de (Postfix) with ESMTP id 14966133C5AA6 for ; Wed, 28 Oct 2009 16:00:47 +0100 (CET) Received: from [95.208.117.111] (helo=frosties.localdomain) by smtp05.web.de with asmtp (TLSv1:AES256-SHA:256) (WEB.DE 4.110 #314) id 1N3A10-0000UK-00 for caml-list@inria.fr; Wed, 28 Oct 2009 16:00:46 +0100 Received: from mrvn by frosties.localdomain with local (Exim 4.69) (envelope-from ) id 1N3A10-0007Aj-Hg for caml-list@inria.fr; Wed, 28 Oct 2009 16:00:46 +0100 From: Goswin von Brederlow To: caml-list@inria.fr Subject: Re: [Caml-list] Re: How to read different ints from a Bigarray? References: <87eiond3of.fsf@frosties.localdomain> Date: Wed, 28 Oct 2009 16:00:46 +0100 In-Reply-To: (Sylvain Le Gall's message of "Wed, 28 Oct 2009 14:16:52 +0000 (UTC)") Message-ID: <87639zd0m9.fsf@frosties.localdomain> User-Agent: Gnus/5.110006 (No Gnus v0.6) XEmacs/21.4.22 (linux) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Sender: goswin-v-b@web.de X-Sender: goswin-v-b@web.de X-Provags-ID: V01U2FsdGVkX18hZVrGuPG6fPVW1+FhYbGUo5Krt6cCOZFdEe5X DJ1OEJYO5RJ8DLPex6aGb3nfxxrgrX9sAm83ezCCDfoHRG8YnH FdgiQR2+E= X-Spam: no; 0.00; bigarray:01 le-gall:01 buffer:01 elt:01 bigarray:01 buf:01 buf:01 elt:01 endian:01 byte:01 endian:01 elegantly:01 stubs:01 blit:01 memcpy:01 Sylvain Le Gall writes: > Hello, > > On 28-10-2009, Goswin von Brederlow wrote: >> Hi, >> >> I'm working on binding s for linux libaio library (asynchron IO) with >> a sharp eye on efficiency. That means no copying must be done on the >> data, which in turn means I can not use string as buffer type. >> >> The best type for this seems to be a (int, int8_unsigned_elt, >> c_layout) Bigarray.Array1.t. So far so good. >> >> Now I define helper functions: >> >> let get_uint8 buf off = buf.{off} >> let set_uint8 buf off x = buf.{off} <- x >> >> But I want more: >> >> get/set_int8 - do I use Obj.magic to "convert" to int8_signed_elt? >> >> And endian correcting access for larger ints: >> >> get/set_big_uint16 >> get/set_big_int16 >> get/set_little_uint16 >> get/set_little_int16 >> get/set_big_uint24 >> ... >> get/set_little_int56 >> get/set_big_int64 >> get/set_little_int64 >> >> What is the best way there? For uintXX I can get_uint8 each byte and >> shift and add them together. But that feels inefficient as each access >> will range check and the shifting generates a lot of code while cpus >> can usualy endian correct an int more elegantly. >> >> Is it worth the overhead of calling a C function to write optimized >> stubs for this? >> >> And last: >> >> get/set_string, blit_from/to_string >> >> Do I create a string where needed and then loop over every char >> calling s.(i) <- char_of_int buf.{off+i}? Or better a C function using >> memcpy? >> >> What do you think? >> > > Well, we talk about this a little bit, but here is my opinion: > - calling a C function to add a single int will generate a big overhead > - OCaml string are quite fast to modify values > > So to my mind the best option is to have a buffer string (say 16/32 > char) where you put data inside and flush it in a single C call to > Bigarray. > > E.g.: > let append_char t c = > if t.idx >= 64 then > ( > flush t.bigarray t.buffer; > t.idx <- 0 > ); > t.buffer.(t.idx) <- c; > t.idx <- t.idx + 1 > > let append_little_uint16 t i = > append_char t ((i lsr 8) land 0xFF); > append_char t ((i lsr 0) land 0xFF) > > > I have used this kind of technique and it seems as fast as C, and a lot > less C coding. > > Regards, > Sylvain Le Gall This wont work so nicely: - Writes are not always in sequence. I want to do a stream access too where this could be verry effective. But the plain buffer is more for random / known offset access. At a minimum you would have holes for alignment. - It makes read/write buffers complicated as you need to flush or peek the string in case of uncommited changes. I can't do write-only buffers as I want to be able to write a buffer and then add a checksum to it in my application. The lib should not block that. - The data is passed to libaio and needs to be kept alive and unmoved as long as libaio knows it. I was hoping I could use the pointer to the data to register/unregister GC roots without having to add a another custom header and indirections. I also still wonder how bad a C function call really is. Consider the case of writing an int64. Directly: You get one C call that does range check, endian convert and write in one go. Bffered: With your code you have 7 Int64 shifts, 8 Int64 lands, 8 conversions to int, at least one index check (more likely 8 to avoid handling unaligned access) and 1/8 C call to blit the 64 byte buffer string into the Bigarray. MfG Goswin PS: Is a.{i} <- x a C call?