Re: [Caml-list] String Problem

[Olivier Andrieu <andrieu@ijm.jussieu.fr>]

 Thomas Fischbacher [Wed, 9 Jun 2004]:
 > 
 > Dear Caml hackers,
 > 
 > I am just doing some quite large (string theory) calculation which
 > basically runs through a huge tree and does some computation at
 > every node in ocaml which I have to parallelize in an effective
 > way. My present approach is to set an alarm for the process doing
 > the calculation, then splitting into chunks and serializing all the
 > work that corresponds to nodes that have been touched but for which
 > the calculation has not yet been finished. The serialized strings
 > are then compressed and sent out via the net to other machines to
 > help with the calculation.
 > 
 > I'd love to avoid temporary files, as these are not necessary, and
 > my design is simpler and cleaner without having to worry about
 > filesystem issues.
 >
 > Now I encounter the problem that ocaml can only serialize to
 > strings, but these are limited to 16 MB in size. If my data set
 > (which is structured in a complicated way, i.e. it would be quite
 > some effort to write specialized readers and printers) gets large
 > enough, this entire approach therefore breaks down.

It's quite easy to serialize to a Bigarray with a bit of C code
(warning, not tested): 

,----
| #include "intext.h"
| #include "bigarray.h"
| 
| CAMLprim value ml_marshal_to_bigarray(value v, value flags)
| {
|   char *buf;
|   long len;
|   output_value_to_malloc(v, flags, &buf, &len);
|   return alloc_bigarray(BIGARRAY_UINT8 | BIGARRAY_C_LAYOUT | BIGARRAY_MANAGED, 
|                         1, buf, &len);
| }
| 
| CAMLprim value ml_demarshal_from_bigarray(value b)
| {
|   struct caml_bigarray *b_arr = Bigarray_val(b);
|   return input_value_from_block(b_arr->data, b_arr->dim[0]);
| }
`----

,----
| open Bigarray
| 
| external marshal_to_bigarray : 
|   'a -> Marshal.extern_flags list -> 
|   (char, int8_unsigned_elt, c_layout) Array1.t 
|      = "ml_marshal_to_bigarray"
| 
| external demarshal_from_bigarray :
|   (char, int8_unsigned_elt, c_layout) Array1.t -> 'a 
|      = "ml_demarshal_from_bigarray"
`----

Alternatively, buy a 64 bits computer :)

-- 
   Olivier

-------------------
To unsubscribe, mail caml-list-request@inria.fr Archives: http://caml.inria.fr
Bug reports: http://caml.inria.fr/bin/caml-bugs FAQ: http://caml.inria.fr/FAQ/
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners