[Caml-list] camlimages and kernel memory

[Caml-list] camlimages and kernel memory

[Fernando Alegre]

Hi,

We would like to use some kernel memory buffers from the camlimages libraries
without copying the data to userland. The data comes from digital cameras
directly (using DMA) to a kernel ringbuffer. Caml does not own that memory,
but we want Caml to use it as just another Image.t chunk (with a wrapper,
of course.)

Our question is: is there a safe way to do this? We don't want the garbage
collector to mess with the kernel memory, but we don't want to have
to code separately for "userland" and "kernel" images...

Any suggestion will be appreciated.

Thanks,

       Fernando
       
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] camlimages and kernel memory

[Gerd Stolpmann]

On 2002.06.06 01:48 Fernando Alegre wrote:
> 
> Hi,
> 
> We would like to use some kernel memory buffers from the camlimages libraries
> without copying the data to userland. The data comes from digital cameras
> directly (using DMA) to a kernel ringbuffer. Caml does not own that memory,
> but we want Caml to use it as just another Image.t chunk (with a wrapper,
> of course.)
> 
> Our question is: is there a safe way to do this? We don't want the garbage
> collector to mess with the kernel memory, but we don't want to have
> to code separately for "userland" and "kernel" images...
> 
> Any suggestion will be appreciated.

One important information is missing: which operating system?

For example, Linux (but other OS, too) allow userland processes to access the
kernel memory by opening and mmapping the file /dev/kmem. This effects that
kernel memory is mapped into the userland address space. However, mmap is
usually restricted to 4k blocks, i.e. you can mmap only a whole number of
4k blocks.

O'Caml can handle pointers to blocks that are outside of the region that is
managed by its own routines. The blocks must have the right format, of course,
i.e. strings must have the right O'Caml header. I suppose this header is not
in the ringbuffer.

If the ringbuffer happens to begin at a 4k boundary, you have won, because you
can manage that there is such a header without modifying kernel memory, i.e.
if the mmaped buffer begins at address A, just allocate anonymous memory in the
4k block before A and write the needed header into it.

In all other cases, you must modify kernel memory to ensure that there is
a header. I don't know whether this is possible.

The other problem is whether camlimages can handle data that is organized as ring.
I don't have any ideas.

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 45             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] camlimages and kernel memory

[Xavier Leroy]

> For example, Linux (but other OS, too) allow userland processes to access the
> kernel memory by opening and mmapping the file /dev/kmem. This effects that
> kernel memory is mapped into the userland address space. However, mmap is
> usually restricted to 4k blocks, i.e. you can mmap only a whole number of
> 4k blocks.
> 
> O'Caml can handle pointers to blocks that are outside of the region that is
> managed by its own routines. The blocks must have the right format, of course,
> i.e. strings must have the right O'Caml header. I suppose this header is not
> in the ringbuffer.
> 
> If the ringbuffer happens to begin at a 4k boundary, you have won, because you
> can manage that there is such a header without modifying kernel memory, i.e.
> if the mmaped buffer begins at address A, just allocate anonymous memory in the
> 4k block before A and write the needed header into it.

This is the hard way :-)  A simpler way to memory map files or devices
in a Caml program is to use the function "map_file" from the Bigarray
library.  However, it currently always map the file from offset 0,
which is probably not appropriate for /dev/kmem...  I'll have to look
into this limitation.

> The other problem is whether camlimages can handle data that is
> organized as ring.  I don't have any ideas.

The type Image.t is a relatively complex data structure, so some work
is definitely needed to go from a raw string or bigarray
(corresponding to the memory-mapped file) to a value of type Image.t.
I'll let the authors of CamlImages comment on that.

- Xavier Leroy
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] camlimages and kernel memory

[Jun P.FURUSE]

Hello,

> This is the hard way :-)  A simpler way to memory map files or devices
> in a Caml program is to use the function "map_file" from the Bigarray
> library.  However, it currently always map the file from offset 0,
> which is probably not appropriate for /dev/kmem...  I'll have to look
> into this limitation.
> 
> > The other problem is whether camlimages can handle data that is
> > organized as ring.  I don't have any ideas.
> 
> The type Image.t is a relatively complex data structure, so some work
> is definitely needed to go from a raw string or bigarray
> (corresponding to the memory-mapped file) to a value of type Image.t.
> I'll let the authors of CamlImages comment on that.

If the kernel memory contains the same data structure as one of
the camlimages internal image formats, the solution may be... 
Somehow (= I do not know) get the kernel memory as a raw string,
then create an image using ???.create_with function. 

The kernel memory may use different pixel layout, I am afraid.
In such case, you have to write your own version of module like
Rgb24, Index8 and so on. I am sorry but they are not well documented...

BTW, once I tried to implement Image.t using Bigarray, but it was too
slow. It seemed to me that it performed the array boundary checks for
each access. For image manipulation purposes, the unsafe versions of set
and get are really required. 

--
jun
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners

Re: [Caml-list] camlimages and kernel memory

[Fernando Alegre]

Thank you for all the good suggestions. We are currently using Bigarrays and
mmap to read images from files, simulating the dcams. We map the whole
file into a bigarray and use ad-hoc functions to extract the rgb data
without worriying much about headers. This is OK  for now, but we may be
interested in the future in interfacing with the camlimages library.

So, the next question is: Can we use Obj.magic to convert a bigarray to a
string without copying data? Will this trick work   with boundary checks
turned off, so that we can have the advantage of mmap and no bound checks
at the same time?

Thanks again,

       Fernando Alegre
       
On Wed, Jun 12, 2002 at 06:20:37PM +0200, Jun P.FURUSE wrote:

> The kernel memory may use different pixel layout, I am afraid.
> In such case, you have to write your own version of module like
> Rgb24, Index8 and so on. I am sorry but they are not well documented...


> BTW, once I tried to implement Image.t using Bigarray, but it was too
> slow. It seemed to me that it performed the array boundary checks for
> each access. For image manipulation purposes, the unsafe versions of set
> and get are really required. 
-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners