reading/writing binary data

reading/writing binary data

[Dário Abdulrehman]

[-- Attachment #1: Type: text/plain, Size: 808 bytes --]

I am writing a program that encodes data in 32 bit ints.  Ideally I would
like to have 32 bit unsigned ints, but OCaml only has 31 bit signed ints.

I have a large binary file with the data that I will mmap with the
Bigarray.Array1.map_file function.

When I test the following code I am puzzled by the fact that Ocaml doesn't
give me back the int in the same format as I wrote it.

Why doesn't OCaml read back data in the same format?

I need to read the data fast without having to manipulate bits beyond the
minimum decoding to get my data back from the binary representation.

Thanks.

let test_data = open_out_bin "test.data" in
  output_binary_int test_data 1

let data =
  let fd = Unix.openfile "test.data" [Unix.O_RDONLY] 0 in
  Bigarray.Array1.map_file fd Bigarray.int Bigarray.c_layout false (-1)

[-- Attachment #2: Type: text/html, Size: 951 bytes --]

Re: [Caml-list] reading/writing binary data

[Olivier Andrieu]

> I am writing a program that encodes data in 32 bit ints.  Ideally I would
> like to have 32 bit unsigned ints, but OCaml only has 31 bit signed ints.

It also has 32 bits signed ints (and 64 bits).

> I have a large binary file with the data that I will mmap with the
> Bigarray.Array1.map_file function.
>
> When I test the following code I am puzzled by the fact that Ocaml doesn't
> give me back the int in the same format as I wrote it.
>
> Why doesn't OCaml read back data in the same format?

because `output_binary_int' uses a big-endian encoding (cf. the reference
manual) whereas Bigarray.map_file uses the native endianess (probably
little-endian in your case).

Don't use different methods to write your data and read it back !

-- 
  Olivier

Re: [Caml-list] reading/writing binary data

[Dário Abdulrehman]

[-- Attachment #1: Type: text/plain, Size: 1348 bytes --]

On 11/21/06, Olivier Andrieu <oandrieu@nerim.net> wrote:
>
> > I am writing a program that encodes data in 32 bit ints.  Ideally I
> would
> > like to have 32 bit unsigned ints, but OCaml only has 31 bit signed
> ints.
>
> It also has 32 bits signed ints (and 64 bits).


Yes but the type int32 is not as efficient as int and I have huge data file.
>From the reference manual: " Performance notice: values of type int32 occupy
more memory space than values of type int, and arithmetic operations on
int32 are generally slower than those on int. Use int32 only when the
application requires exact 32-bit arithmetic."



> I have a large binary file with the data that I will mmap with the
> > Bigarray.Array1.map_file function.
> >
> > When I test the following code I am puzzled by the fact that Ocaml
> doesn't
> > give me back the int in the same format as I wrote it.
> >
> > Why doesn't OCaml read back data in the same format?
>
> because `output_binary_int' uses a big-endian encoding (cf. the reference
> manual) whereas Bigarray.map_file uses the native endianess (probably
> little-endian in your case).
>
> Don't use different methods to write your data and read it back !


Is there some library for writing binary int (not int32) that writes the
data in little endian (I think map_file reads it back in little endian on my
processor) ?

[-- Attachment #2: Type: text/html, Size: 2013 bytes --]

Re: [Caml-list] reading/writing binary data

[Dmitry Bely]

On 11/21/06, Dário Abdulrehman <drehman31@gmail.com> wrote:

> Is there some library for writing binary int (not int32) that writes the
> data in little endian (I think map_file reads it back in little endian on my
> processor) ?

Yes, sure - the library is below :)

let output_binary_int_little_endian ch n =
  output_byte ch (n land 0xff);
  output_byte ch ((n lsr 8) land 0xff);
  output_byte ch ((n lsr 16) land 0xff);
  output_byte ch ((n lsr 24) land 0xff)

- Dmitry Bely

Re: [Caml-list] reading/writing binary data

[Jon Harrop]

On Tuesday 21 November 2006 13:56, Dmitry Bely wrote:
> let output_binary_int_little_endian ch n =
>   output_byte ch (n land 0xff);
>   output_byte ch ((n lsr 8) land 0xff);
>   output_byte ch ((n lsr 16) land 0xff);
>   output_byte ch ((n lsr 24) land 0xff)

I want to see those HOFs:

  let output_binary_int_little_endian ch n =
    List.iter (fun i -> output_byte ch ((n lsr i) land 0xff)) [0; 8; 16; 24];;

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
Objective CAML for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists

Re: [Caml-list] reading/writing binary data

[Florian Hars]

Jon Harrop schrieb:
> I want to see those HOFs:
> 
>   let output_binary_int_little_endian ch n =
>     List.iter (fun i -> output_byte ch ((n lsr i) land 0xff)) [0; 8; 16; 24];;

Why not go the whole way?

let little_endian = [0; 8; 16; 24]
let middle_endian = [16; 24; 0; 8]
let big_endian = [24; 16; 8; 0]

let output_binary_int endianness ch n =
   List.iter (fun i -> output_byte ch ((n lsr i) land 0xff)) endianness

Now all we need is a PDP-11 port...

Yours, Florian.