[Caml-list] Simple compression library

[Caml-list] Simple compression library

[Dhek Uir]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

I'm looking for a compression library such as camlzip to read/write
compressed files (any lossless compression algorithm is fine), but having a
function similar to Pervasives.output_value, which does not seem to be
available in camlzip (and I do not see how I could easily implement it).

Would it be simple to modify camlzip to add this function? I believe that
if this were the case, then it would already have been done, but I might be
wrong.

Otherwise, is there another OCaml library with an API very similar to
Pervasives'? I'd prefer a lightweight solution (avoiding Core and
Batteries) if possible.

[-- Attachment #2: Type: text/html, Size: 679 bytes --]

Re: [Caml-list] Simple compression library

[Jacques-Henri Jourdan]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

Le 02/02/2015 16:39, Dhek Uir a écrit :
> I'm looking for a compression library such as camlzip to read/write
> compressed files (any lossless compression algorithm is fine), but
> having a function similar to Pervasives.output_value, which does not
> seem to be available in camlzip (and I do not see how I could easily
> implement it).
>
> Would it be simple to modify camlzip to add this function? I believe
> that if this were the case, then it would already have been done, but
> I might be wrong.
>
> Otherwise, is there another OCaml library with an API very similar to
> Pervasives'? I'd prefer a lightweight solution (avoiding Core and
> Batteries) if possible.

Can't you do Marshal.to_string, and then use Camlzip on the generated
string ?

-- 
JH Jourdan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Caml-list] Simple compression library

[Dhek Uir]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On Mon, Feb 2, 2015 at 4:43 PM, Jacques-Henri Jourdan <
jacques-henri.jourdan@inria.fr> wrote:

> Le 02/02/2015 16:39, Dhek Uir a écrit :
> > I'm looking for a compression library such as camlzip to read/write
> > compressed files (any lossless compression algorithm is fine), but
> > having a function similar to Pervasives.output_value, which does not
> > seem to be available in camlzip (and I do not see how I could easily
> > implement it).
> >
> > Would it be simple to modify camlzip to add this function? I believe
> > that if this were the case, then it would already have been done, but
> > I might be wrong.
> >
> > Otherwise, is there another OCaml library with an API very similar to
> > Pervasives'? I'd prefer a lightweight solution (avoiding Core and
> > Batteries) if possible.
>
> Can't you do Marshal.to_string, and then use Camlzip on the generated
> string ?
>
>
Indeed, I can. I believe the hard part is actually the input_value
operation.
But I haven't looked enough into Marshal, maybe my solution can be found
there.

[-- Attachment #2: Type: text/html, Size: 1600 bytes --]

Re: [Caml-list] Simple compression library

[Drup]

[-- Attachment #1: Type: text/plain, Size: 1622 bytes --]

input/output_value are using Marshal under the hood, so it's going to be 
the same.

However, be aware that marshall (and the _value functions) are quite 
unsafe. If you have an error in the data (or your change the 
representation in OCaml), you will have ... surprising errors (including 
segfaults).

There are plenty of safer solutions for serialization (deriving, sexp, 
yojson, to name a few).

Le 02/02/2015 17:05, Dhek Uir a écrit :
>
>
> On Mon, Feb 2, 2015 at 4:43 PM, Jacques-Henri Jourdan 
> <jacques-henri.jourdan@inria.fr 
> <mailto:jacques-henri.jourdan@inria.fr>> wrote:
>
>     Le 02/02/2015 16:39, Dhek Uir a écrit :
>     > I'm looking for a compression library such as camlzip to read/write
>     > compressed files (any lossless compression algorithm is fine), but
>     > having a function similar to Pervasives.output_value, which does not
>     > seem to be available in camlzip (and I do not see how I could easily
>     > implement it).
>     >
>     > Would it be simple to modify camlzip to add this function? I believe
>     > that if this were the case, then it would already have been
>     done, but
>     > I might be wrong.
>     >
>     > Otherwise, is there another OCaml library with an API very
>     similar to
>     > Pervasives'? I'd prefer a lightweight solution (avoiding Core and
>     > Batteries) if possible.
>
>     Can't you do Marshal.to_string, and then use Camlzip on the generated
>     string ?
>
> Indeed, I can. I believe the hard part is actually the input_value 
> operation.
> But I haven't looked enough into Marshal, maybe my solution can be 
> found there.


[-- Attachment #2: Type: text/html, Size: 3211 bytes --]

Re: [Caml-list] Simple compression library

[Dhek Uir]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

But since camlzip does not have an input_value function, it seems I must
set up a buffer to decompress the data, then use Marshal to obtain input
data from the decompressed buffer. But if I use Marshal.from_string to read
the decompressed buffer, then I need to keep track of its position (which
is not returned by Marshal.from_string), and check whether the buffer is
nearly empty, so that I may need to re-read from the compressed in_channel,
etc, etc. Did I miss the right function again?

It seems that Netgzip would require some similar amount of effort, from
what I'm seeing (I see no "channel -> 'a" functions in the API).

Thanks for the serialization tips, though I don't care much about the
safety, I have strong assumptions about the how the data will be produced
and consumed.


On Mon, Feb 2, 2015 at 5:48 PM, Drup <drupyog+caml@zoho.com> wrote:

>  input/output_value are using Marshal under the hood, so it's going to be
> the same.
>
> However, be aware that marshall (and the _value functions) are quite
> unsafe. If you have an error in the data (or your change the representation
> in OCaml), you will have ... surprising errors (including segfaults).
>
> There are plenty of safer solutions for serialization (deriving, sexp,
> yojson, to name a few).
>
> Le 02/02/2015 17:05, Dhek Uir a écrit :
>
>
>
> On Mon, Feb 2, 2015 at 4:43 PM, Jacques-Henri Jourdan <
> jacques-henri.jourdan@inria.fr> wrote:
>
>> Le 02/02/2015 16:39, Dhek Uir a écrit :
>>  > I'm looking for a compression library such as camlzip to read/write
>> > compressed files (any lossless compression algorithm is fine), but
>> > having a function similar to Pervasives.output_value, which does not
>> > seem to be available in camlzip (and I do not see how I could easily
>> > implement it).
>> >
>> > Would it be simple to modify camlzip to add this function? I believe
>> > that if this were the case, then it would already have been done, but
>> > I might be wrong.
>> >
>> > Otherwise, is there another OCaml library with an API very similar to
>> > Pervasives'? I'd prefer a lightweight solution (avoiding Core and
>> > Batteries) if possible.
>>
>>  Can't you do Marshal.to_string, and then use Camlzip on the generated
>> string ?
>>
>>
> Indeed, I can. I believe the hard part is actually the input_value
> operation.
>  But I haven't looked enough into Marshal, maybe my solution can be found
> there.
>
>
>

[-- Attachment #2: Type: text/html, Size: 4254 bytes --]

RE: [Caml-list] Simple compression library

[David Allsopp]

Dhek Uir wrote:
> But since camlzip does not have an input_value function, it seems I must set up
> a buffer to decompress the data, then use Marshal to obtain input data from the
> decompressed buffer. But if I use Marshal.from_string to read the decompressed
> buffer, then I need to keep track of its position (which is not returned by 
> Marshal.from_string), and check whether the buffer is nearly empty, so that I 
> may need to re-read from the compressed in_channel, etc, etc. Did I miss the 
> right function again?

Marshal has everything you need:

let input_value ch =
  let header = String.create Marshal.header_size
  in
    Gzip.really_input ch header 0 Marshal.header_size;
    let buffer = String.create (Marshal.total_size header 0)
    in
      Gzip.really_input ch buffer Marshal.header_size (Marshal.data_size header 0);
      String.unsafe_blit header 0 buffer 0 Marshal.header_size;
      Marshal.from_string buffer 0


David

Re: [Caml-list] Simple compression library

[Dhek Uir]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

Wow, I completely overlooked that. I was convinced that
Gzip.input/Gzip.really_input returned the number of *compressed* characters
read. I misread "uncompresses up to len characters" as "advances the
compressed buffer up to len characters". The phrase "[Gzip.input] returns
the actual number of characters read" didn't help me either.

Anyway, thanks a lot, it's much simpler than I expected!




On Mon, Feb 2, 2015 at 6:38 PM, David Allsopp <dra-news@metastack.com>
wrote:

> Dhek Uir wrote:
> > But since camlzip does not have an input_value function, it seems I must
> set up
> > a buffer to decompress the data, then use Marshal to obtain input data
> from the
> > decompressed buffer. But if I use Marshal.from_string to read the
> decompressed
> > buffer, then I need to keep track of its position (which is not returned
> by
> > Marshal.from_string), and check whether the buffer is nearly empty, so
> that I
> > may need to re-read from the compressed in_channel, etc, etc. Did I miss
> the
> > right function again?
>
> Marshal has everything you need:
>
> let input_value ch =
>   let header = String.create Marshal.header_size
>   in
>     Gzip.really_input ch header 0 Marshal.header_size;
>     let buffer = String.create (Marshal.total_size header 0)
>     in
>       Gzip.really_input ch buffer Marshal.header_size (Marshal.data_size
> header 0);
>       String.unsafe_blit header 0 buffer 0 Marshal.header_size;
>       Marshal.from_string buffer 0
>
>
> David
>

[-- Attachment #2: Type: text/html, Size: 2031 bytes --]