From mboxrd@z Thu Jan 1 00:00:00 1970 Received: (from majordomo@localhost) by pauillac.inria.fr (8.7.6/8.7.3) id KAA22244; Mon, 2 Aug 2004 10:00:46 +0200 (MET DST) X-Authentication-Warning: pauillac.inria.fr: majordomo set sender to owner-caml-list@pauillac.inria.fr using -f Received: from nez-perce.inria.fr (nez-perce.inria.fr [192.93.2.78]) by pauillac.inria.fr (8.7.6/8.7.3) with ESMTP id KAA23692 for ; Mon, 2 Aug 2004 10:00:45 +0200 (MET DST) Received: from will.iki.fi (will.iki.fi [217.169.64.20]) by nez-perce.inria.fr (8.12.10/8.12.10) with ESMTP id i7280iEV007174 for ; Mon, 2 Aug 2004 10:00:44 +0200 Received: from [10.0.20.162] (fa-3-0-0.fw.exomi.com [217.169.64.99]) by will.iki.fi (Postfix) with ESMTP id 5C20B24; Mon, 2 Aug 2004 11:00:43 +0300 (EEST) Message-ID: <410DF4A8.8090306@exomi.com> Date: Mon, 02 Aug 2004 11:00:40 +0300 From: Ville-Pertti Keinonen User-Agent: Mozilla Thunderbird 0.7.1 (X11/20040708) X-Accept-Language: en-us, en MIME-Version: 1.0 To: "Brandon J. Van Every" Cc: caml Subject: Re: [Caml-list] Wish List for Large Mutable Objects References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Miltered: at nez-perce with ID 410DF4AC.000 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! X-Loop: caml-list@inria.fr X-Spam: no; 0.00; caml-list:01 brandon:99 api:01 mcclain:01 mappings:01 aligned:01 passing:01 aligned:01 mappings:01 natively:01 bigarrays:01 higher-level:01 abstractions:01 higher-level:01 abstractions:01 Sender: owner-caml-list@pauillac.inria.fr Precedence: bulk Brandon J. Van Every wrote: >loop optimizations for 3d graphics. That said, Scatter-Gather DMA is >generally a property of a memory controller, i.e. a chipset on a >motherboard. I seriously doubt you have user mode access to such memory >controllers. If you do, point me at the API for it. I'm happy to stand >corrected, but as far as I know, scatter-gather DMA is kernel mode stuff >on all common architectures. > > Often things like the readv(2)/writev(2) interface are referred to as "scatter-gather". It just means I/O on regions of memory that aren't contiguous in a single operation. I'm not sure what David McClain is referring to - but I think it's the ability for an "array" to provide another level of virtualization so that the underlying data needn't be contiguous in the address space of the process. That seems a bit excessive - a more limited part of what he's suggesting could be more reasonable - being able to map a part of a file, from an arbitrary offset, as a Bigarray could be useful. Even this would require some separation of storage management from the actual "Bigarray header", since operating systems require the underlying mappings to be page aligned. I suspect this could be as simple as passing a page-truncated offset to mmap(2), adding the remaining offset to the returned address and page-truncating the address passed to munmap(2). This doesn't address the problem with most CPUs requiring the actual objects to be aligned, for which adding an "offset" between the beginning of the mapping and the beginning of the visible array isn't sufficient if the mapped file doesn't align things appropriately for the CPU (for arbitrary file formats, there's also endianness issues to consider). Using subarrays instead of offsetted memory mappings protects against this, and makes offsetted mappings "unnecessary" altogether, but obviously if the file is big enough, the waste of address space due to the extra mappings can be significant on 32-bit systems...which as I understand was part of the original problem. Typing this on an Athlon 64 and sitting next to an Alpha, such things seem like legacy issues to me, especially since OCaml supports both architectures natively. The main issue I have with the suggestions regarding turning Bigarrays into higher-level abstractions altogether is that it would make them considerably less efficient. The higher-level abstractions can always be implemented as layers on top of the current abstractions, which in my opinion is the right approach. Of course it isn't my decision in any way. ------------------- To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/ Beginner's list: http://groups.yahoo.com/group/ocaml_beginners