Re: [Caml-list] marshaling limits

["Markus Mottl" <markus.mottl@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1858 bytes --]

On 1/17/07, Yaron Minsky <yminsky@cs.cornell.edu> wrote:
>
> Don't quote me on this, but I believe that marshal uses a string in
> bytecode with threads, uses straight malloc with bytecode and no threads,
> and never uses strings in native code.  I'm /very/ unsure about that last
> one, but I am pretty confident that in some cases, whether it uses strings
> depends on whether threads are involved.
>

I think the question is more along the lines "byte code threads" vs. native
(e.g. POSIX) threads rather than "byte vs. native code".  It's true that
byte code threads, which can naturally only be used with byte code, require
an intermediate copy step to OCaml-strings if you want to write to
channels.  That's bad on 32bit platforms due to the size limitations on
strings (< 16MB).

I'd recommend using Bigarrays of characters to marshal out data in cases
where OCaml-strings don't suffice.  The code for this is extremely simple:

  extern CAMLprim int
  caml_output_value_to_block(value v, value v_flags, char *bstr, int len);

  CAMLprim value bigstring_marshal_stub(value v, value v_flags)
  {
    char *buf;
    long len;
    int alloc_flags = BIGARRAY_UINT8 | BIGARRAY_C_LAYOUT | BIGARRAY_MANAGED;
    caml_output_value_to_malloc(v, v_flags, &buf, &len);
    return alloc_bigarray(alloc_flags, 1, buf, &len);
  }

The signature of the OCaml-function is:

  external marshal : 'a -> Marshal.extern_flags list -> t  =
"bigstring_marshal_stub"

Where type "t" is a bigarray of characters with C-layout.

You can even do without the intermediate copying if you know the maximum
size of the marshalled data and preallocate a bigarray for that.  Use
"caml_output_value_to_block" for that purpose.  It's defined in
"byterun/extern.c" of the OCaml-distribution.

Regards,
Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com

[-- Attachment #2: Type: text/html, Size: 2620 bytes --]