From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on yquem.inria.fr X-Spam-Level: * X-Spam-Status: No, score=1.9 required=5.0 tests=AWL,HTML_MESSAGE,INFO_TLD, SPF_NEUTRAL autolearn=disabled version=3.1.3 X-Original-To: caml-list@yquem.inria.fr Delivered-To: caml-list@yquem.inria.fr Received: from discorde.inria.fr (discorde.inria.fr [192.93.2.38]) by yquem.inria.fr (Postfix) with ESMTP id C97EFBC0B for ; Wed, 17 Jan 2007 23:51:40 +0100 (CET) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.169]) by discorde.inria.fr (8.13.6/8.13.6) with ESMTP id l0HMpeD5026589 for ; Wed, 17 Jan 2007 23:51:40 +0100 Received: by ug-out-1314.google.com with SMTP id q2so12718uge for ; Wed, 17 Jan 2007 14:51:38 -0800 (PST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=NewrgixRYkws3xYLcrbMyWEjyzezEa9viMXRqM3CNFKMTymKpw7xNoz2rkkeB2vOfnugKTbjDKn2Yp9fTVdihNcu12xQRp+dFiehJxmMBfTAaJGE052fmHG523GzP7nrs4YU0LrRCDFiHD4ug06JQ3klLpcwYD4T0MYClzILPWY= Received: by 10.82.138.6 with SMTP id l6mr35331bud.1169074297769; Wed, 17 Jan 2007 14:51:37 -0800 (PST) Received: by 10.82.107.11 with HTTP; Wed, 17 Jan 2007 14:51:37 -0800 (PST) Message-ID: Date: Wed, 17 Jan 2007 17:51:37 -0500 From: "Markus Mottl" To: "Yaron Minsky" Subject: Re: [Caml-list] marshaling limits Cc: "Sebastien Ferre" , caml-list@yquem.inria.fr, "=?ISO-8859-1?Q?Daniel_B=FCnzli?=" In-Reply-To: <891bd3390701171150m601bb382ufc1d05f088281b93@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_174732_25554261.1169074297500" References: <2a1a1a0c0701161248l5a1d9ad9q739da9c593fc5219@mail.gmail.com> <45ADE8FE.5070004@irisa.fr> <45AE41C7.8080409@univ-paris12.fr> <45AE43A9.3090904@irisa.fr> <45AE5001.2080709@irisa.fr> <891bd3390701171150m601bb382ufc1d05f088281b93@mail.gmail.com> X-j-chkmail-Score: MSGID : 45AEA87C.001 on discorde : j-chkmail score : X : 0/20 1 0.000 -> 1 X-Miltered: at discorde with ID 45AEA87C.001 by Joe's j-chkmail (http://j-chkmail . ensmp . fr)! X-Spam: no; 0.00; markus:01 mottl:01 markus:01 mottl:01 marshaling:01 yaron:01 minsky:01 yminsky:01 bytecode:01 threads:01 malloc:01 bytecode:01 threads:01 byte:01 posix:01 ------=_Part_174732_25554261.1169074297500 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline On 1/17/07, Yaron Minsky wrote: > > Don't quote me on this, but I believe that marshal uses a string in > bytecode with threads, uses straight malloc with bytecode and no threads, > and never uses strings in native code. I'm /very/ unsure about that last > one, but I am pretty confident that in some cases, whether it uses strings > depends on whether threads are involved. > I think the question is more along the lines "byte code threads" vs. native (e.g. POSIX) threads rather than "byte vs. native code". It's true that byte code threads, which can naturally only be used with byte code, require an intermediate copy step to OCaml-strings if you want to write to channels. That's bad on 32bit platforms due to the size limitations on strings (< 16MB). I'd recommend using Bigarrays of characters to marshal out data in cases where OCaml-strings don't suffice. The code for this is extremely simple: extern CAMLprim int caml_output_value_to_block(value v, value v_flags, char *bstr, int len); CAMLprim value bigstring_marshal_stub(value v, value v_flags) { char *buf; long len; int alloc_flags = BIGARRAY_UINT8 | BIGARRAY_C_LAYOUT | BIGARRAY_MANAGED; caml_output_value_to_malloc(v, v_flags, &buf, &len); return alloc_bigarray(alloc_flags, 1, buf, &len); } The signature of the OCaml-function is: external marshal : 'a -> Marshal.extern_flags list -> t = "bigstring_marshal_stub" Where type "t" is a bigarray of characters with C-layout. You can even do without the intermediate copying if you know the maximum size of the marshalled data and preallocate a bigarray for that. Use "caml_output_value_to_block" for that purpose. It's defined in "byterun/extern.c" of the OCaml-distribution. Regards, Markus -- Markus Mottl http://www.ocaml.info markus.mottl@gmail.com ------=_Part_174732_25554261.1169074297500 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline On 1/17/07, Yaron Minsky <yminsky@cs.cornell.edu> wrote:
Don't quote me on this, but I believe that marshal uses a string in bytecode with threads, uses straight malloc with bytecode and no threads, and never uses strings in native code.  I'm /very/ unsure about that last one, but I am pretty confident that in some cases, whether it uses strings depends on whether threads are involved.

I think the question is more along the lines "byte code threads" vs. native (e.g. POSIX) threads rather than "byte vs. native code".  It's true that byte code threads, which can naturally only be used with byte code, require an intermediate copy step to OCaml-strings if you want to write to channels.  That's bad on 32bit platforms due to the size limitations on strings (< 16MB).

I'd recommend using Bigarrays of characters to marshal out data in cases where OCaml-strings don't suffice.  The code for this is extremely simple:

  extern CAMLprim int
  caml_output_value_to_block(value v, value v_flags, char *bstr, int len);

  CAMLprim value bigstring_marshal_stub(value v, value v_flags)
  {
    char *buf;
    long len;
    int alloc_flags = BIGARRAY_UINT8 | BIGARRAY_C_LAYOUT | BIGARRAY_MANAGED;
    caml_output_value_to_malloc(v, v_flags, &buf, &len);
    return alloc_bigarray(alloc_flags, 1, buf, &len);
  }

The signature of the OCaml-function is:

  external marshal : 'a -> Marshal.extern_flags list -> t  = "bigstring_marshal_stub"

Where type "t" is a bigarray of characters with C-layout.

You can even do without the intermediate copying if you know the maximum size of the marshalled data and preallocate a bigarray for that.  Use "caml_output_value_to_block" for that purpose.  It's defined in "byterun/extern.c" of the OCaml-distribution.

Regards,
Markus

--
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com ------=_Part_174732_25554261.1169074297500--