Re: [Caml-list] Marshalling data format deteriorates compressibility

[Martin Jambon <ocaml@martin_jambon.emailuser.net>]

[to the list admin: why can I send messages with this subscription and not
  with martin_jambon@emailuser.net?]

On Tue, 27 Jun 2006, Markus Mottl wrote:

> We finally found out what causes the problem: OCaml represents
> pointers to shared data values using relative offsets instead of
> absolute positions within the marshalled data.  This means that e.g.
> an array containing pointers to these values will be represented by a
> sequence of increasing relative offsets, which essentially renders it
> almost incompressible to the usual compression algorithms.
> 
> As it seems, the current marshalling algorithm uses this relative
> addressing approach to save space: relative offsets are encoded with
> variable length (this assumes some degree of locality), which is not
> possible with absolute addressing.  Unfortunately, this does not take
> compression algorithms into account, which may greatly benefit from
> repeating patterns of pointers.

Maybe you can convert the data into a marshal-optimized format before 
marshalling, where you put your shared data into an array, and substitute the 
pointers by array indices, e.g.
   type t = string list
becomes:
   type marshalled_t = (string array * int list)

["a"; "b"; "a"; "a"] -> ([| "a"; "b" |], [0; 1; 0; 0])

That seems like a lot of work, but it shouldn't be too hard to maintain.

By the way, floats don't compress very well. Rounding them as much as possible 
used to save me about 50% of space.


Martin

--
Martin Jambon, PhD
http://martin.jambon.free.fr