[rust-dev] Reading numbers from a mmap

[rust-dev] Reading numbers from a mmap

[Matt]

Sorry if this has been covered before, and also that I'm a complete noob, but I'm thinking about trying a first project in Rust, and trying to learn enough to get started. 

My plan is for an on-disk key-value store. I'm going to end up writing arrays of numbers to disk, and then needing to efficiently read them out of a mmap-ed file. So I'm wondering how in Rust you efficiently/zero-copy-ly take some slice of a read-only mmap and treat it as e.g. a vector of ints?

I see Vec.from_raw_parts() and Vec.from_raw_buf(), but I don't really understand the difference between them, and also they seem like they just give you a vector of the pointer's type, so I don't know how you use them convert the u8s you get from MemoryMap.data() into a vector of a different type, e.g. 32 bit ints.

It seems like there should be a higher level API for this kind of thing, where "casting" a slice of a read-only memory buffer into an immutable vector is not an unsafe operation (I mean, you can do that in Python ;) Either I don't see it in the docs, or it doesn't exist yet; just wondering which :)

Thanks!

Matt

Re: [rust-dev] Reading numbers from a mmap

[Daniel Micay]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On 10/12/14 01:08 PM, Matt wrote:
> Sorry if this has been covered before, and also that I'm a complete noob, but I'm thinking about trying a first project in Rust, and trying to learn enough to get started. 
> 
> My plan is for an on-disk key-value store. I'm going to end up writing arrays of numbers to disk, and then needing to efficiently read them out of a mmap-ed file. So I'm wondering how in Rust you efficiently/zero-copy-ly take some slice of a read-only mmap and treat it as e.g. a vector of ints?
> 
> I see Vec.from_raw_parts() and Vec.from_raw_buf(), but I don't really understand the difference between them, and also they seem like they just give you a vector of the pointer's type, so I don't know how you use them convert the u8s you get from MemoryMap.data() into a vector of a different type, e.g. 32 bit ints.
> 
> It seems like there should be a higher level API for this kind of thing, where "casting" a slice of a read-only memory buffer into an immutable vector is not an unsafe operation (I mean, you can do that in Python ;) Either I don't see it in the docs, or it doesn't exist yet; just wondering which :)
> 
> Thanks!
> 
> Matt

Keep in mind that the file won't be portable if you do this. It's why
it's not a common pattern.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [rust-dev] Reading numbers from a mmap

[Erick Tryzelaar]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

You could try the Cap'n Proto bindings
https://github.com/dwrensha/capnproto-rust, which supports this zero-copy
pattern, and should be fast enough to saturate most disks.




On Wed, Dec 10, 2014 at 10:13 AM, Daniel Micay <danielmicay@gmail.com>
wrote:

> On 10/12/14 01:08 PM, Matt wrote:
> > Sorry if this has been covered before, and also that I'm a complete
> noob, but I'm thinking about trying a first project in Rust, and trying to
> learn enough to get started.
> >
> > My plan is for an on-disk key-value store. I'm going to end up writing
> arrays of numbers to disk, and then needing to efficiently read them out of
> a mmap-ed file. So I'm wondering how in Rust you efficiently/zero-copy-ly
> take some slice of a read-only mmap and treat it as e.g. a vector of ints?
> >
> > I see Vec.from_raw_parts() and Vec.from_raw_buf(), but I don't really
> understand the difference between them, and also they seem like they just
> give you a vector of the pointer's type, so I don't know how you use them
> convert the u8s you get from MemoryMap.data() into a vector of a different
> type, e.g. 32 bit ints.
> >
> > It seems like there should be a higher level API for this kind of thing,
> where "casting" a slice of a read-only memory buffer into an immutable
> vector is not an unsafe operation (I mean, you can do that in Python ;)
> Either I don't see it in the docs, or it doesn't exist yet; just wondering
> which :)
> >
> > Thanks!
> >
> > Matt
>
> Keep in mind that the file won't be portable if you do this. It's why
> it's not a common pattern.
>
>
> _______________________________________________
> Rust-dev mailing list
> Rust-dev@mozilla.org
> https://mail.mozilla.org/listinfo/rust-dev
>
>

[-- Attachment #2: Type: text/html, Size: 2348 bytes --]

Re: [rust-dev] Reading numbers from a mmap

[Simon Sapin]

On 10/12/14 18:08, Matt wrote:
> Sorry if this has been covered before, and also that I'm a complete
> noob, but I'm thinking about trying a first project in Rust, and
> trying to learn enough to get started.
>
> My plan is for an on-disk key-value store. I'm going to end up
> writing arrays of numbers to disk, and then needing to efficiently
> read them out of a mmap-ed file. So I'm wondering how in Rust you
> efficiently/zero-copy-ly take some slice of a read-only mmap and
> treat it as e.g. a vector of ints?
>
> I see Vec.from_raw_parts() and Vec.from_raw_buf(), but I don't really
> understand the difference between them, and also they seem like they
> just give you a vector of the pointer's type, so I don't know how you
> use them convert the u8s you get from MemoryMap.data() into a vector
> of a different type, e.g. 32 bit ints.
>
> It seems like there should be a higher level API for this kind of
> thing, where "casting" a slice of a read-only memory buffer into an
> immutable vector is not an unsafe operation (I mean, you can do that
> in Python ;) Either I don't see it in the docs, or it doesn't exist
> yet; just wondering which :)

Vec is probably not what you want, as it owns its memory and frees it 
when going out of scope. Use a slice instead. Assuming that’s what you 
get from mmap, you can create a slice from a raw pointer and a length:

http://doc.rust-lang.org/std/slice/fn.from_raw_mut_buf.html
http://doc.rust-lang.org/std/slice/fn.from_raw_buf.html

Something like (untested):

   use std::slice::from_raw_mut_buf;
   use std::mem::size_of;
   use libc::c_void;

   // Whatever you do for mmap:
   let (void_pointer, byte_length): (*mut c_void, uint) = ...;

   let i32_pointer: *mut i32 = void_pointer as *mut i32;
   let i32_length: uint = byte_length / size_of::<i32>();
   let slice: &mut [i32] = from_raw_mut_buf(&i32_pointer, i32_length);

Then you can index into `slice`. You’re responsible for making sure that 
the mmap lives at least as long as `slice` does. (Its lifetime it tied 
to that of `i32_pointer`.) Most type annotations can probably be 
inferred, I added them to show what’s going on.

-- 
Simon Sapin

Re: [rust-dev] Reading numbers from a mmap

[David Henningsson]

On 2014-12-10 19:08, Matt wrote:
> Sorry if this has been covered before, and also that I'm a complete noob, but I'm thinking about trying a first project in Rust, and trying to learn enough to get started.
>
> My plan is for an on-disk key-value store. I'm going to end up writing arrays of numbers to disk, and then needing to efficiently read them out of a mmap-ed file. So I'm wondering how in Rust you efficiently/zero-copy-ly take some slice of a read-only mmap and treat it as e.g. a vector of ints?
>
> I see Vec.from_raw_parts() and Vec.from_raw_buf(), but I don't really understand the difference between them, and also they seem like they just give you a vector of the pointer's type, so I don't know how you use them convert the u8s you get from MemoryMap.data() into a vector of a different type, e.g. 32 bit ints.
>
> It seems like there should be a higher level API for this kind of thing, where "casting" a slice of a read-only memory buffer into an immutable vector is not an unsafe operation (I mean, you can do that in Python ;) Either I don't see it in the docs, or it doesn't exist yet; just wondering which :)

Maybe this function will help you:

http://doc.rust-lang.org/std/os/struct.MemoryMap.html

At least on Linux you should be able to open a file, get its fd using 
as_raw_fd() and pass that into the call to MemoryMap::new (see 
MapOption::MapFd()).

// David