CamlMPI: sending marshalled objects

CamlMPI: sending marshalled objects

[Thomas Fischbacher]

Hello everybody,

there is an old Caml/MPI interface, written by Xavier himself in 1998.
This provides a pair of C functions caml_mpi_send/caml_mpi_receive,
which may look a bit funny considering present day ML-C-interface coding
conventions, but GC-wise is actually valid code as far as I can judge.
For reference, I added them at the end of this email.

As far as I can see, output_value_to_malloc() actually should be able to
serialize virtually any ML object (which does not contain alloc_custom()
blocks and similar stuff) to a buffer. Sending this over the net also
should not be much of a problem, regardless of the data being sent. If,
on the other hand, I manually serialized to a string and tried to send
that, I would be bound by the 16 MB maximal string length limitation on
32-bit architectures.

So much the theory. But what I actually find in reality is that -- even
using caml_mpi_send/caml_mpi_receive rather than intermediate strings --
my code crashes when I try to pass around serialized ML data over MPI
which is larger than 16 MB. I am not yet 100% sure that this really is
the problem, but at present, I have every reason to believe so.

So, (1) what is going on here, and (2) how can I repair it?
Is there a quick fix?


value caml_mpi_send(value data, value flags,
                     value dest, value tag, value vcomm)
{
   MPI_Comm comm = Comm_val(vcomm);
   char * buffer;
   long len;

   Begin_root(vcomm)             /* prevent deallocation of communicator */
     output_value_to_malloc(data, flags, &buffer, &len);
     /* This also allocates the buffer */
     enter_blocking_section();
     MPI_Send(buffer, len, MPI_BYTE, Int_val(dest), Int_val(tag), comm);
     leave_blocking_section();
   End_roots();
   stat_free(buffer);
   return Int_val(len);
}


value caml_mpi_receive(value vlen, value source, value tag, value vcomm)
{
   MPI_Comm comm = Comm_val(vcomm);
   mlsize_t len = Long_val(vlen);
   char * buffer;
   MPI_Status status;
   value res;

   Begin_root(vcomm)             /* prevent deallocation of communicator */
     buffer = stat_alloc(len);
     enter_blocking_section();
     MPI_Recv(buffer, len, MPI_BYTE,
              Int_val(source), Int_val(tag), comm, &status);
     leave_blocking_section();
     res = input_value_from_malloc(buffer, 0);
     /* This also deallocates the buffer */
   End_roots();
   return res;
}

-- 
best regards,
Thomas Fischbacher
t.fischbacher@soton.ac.uk

Re: [Caml-list] CamlMPI: sending marshalled objects

[Olivier Andrieu]

Hi,

On Jan 22, 2008 7:52 PM, Thomas Fischbacher <tf@functionality.de> wrote:
>
> Hello everybody,
>
> there is an old Caml/MPI interface, written by Xavier himself in 1998.
> This provides a pair of C functions caml_mpi_send/caml_mpi_receive,
> which may look a bit funny considering present day ML-C-interface coding
> conventions, but GC-wise is actually valid code as far as I can judge.
> For reference, I added them at the end of this email.

I've been using this interface some time ago (in 2006). I rewrote it a
bit to bring it up-to-date with the current conventions.

> As far as I can see, output_value_to_malloc() actually should be able to
> serialize virtually any ML object (which does not contain alloc_custom()
> blocks and similar stuff) to a buffer.

some custom blocks such as Bigarrays can be serialised too.

> Sending this over the net also
> should not be much of a problem, regardless of the data being sent. If,
> on the other hand, I manually serialized to a string and tried to send
> that, I would be bound by the 16 MB maximal string length limitation on
> 32-bit architectures.

right.

> So much the theory. But what I actually find in reality is that -- even
> using caml_mpi_send/caml_mpi_receive rather than intermediate strings --
> my code crashes when I try to pass around serialized ML data over MPI
> which is larger than 16 MB. I am not yet 100% sure that this really is
> the problem, but at present, I have every reason to believe so.
>
> So, (1) what is going on here, and (2) how can I repair it?
> Is there a quick fix?

I'm not sure what is going on, I was able to transmit large marshalled
values IIRC.

One serious problem in the current code is the treatment of errors.
Ocamlmpi registers an error handler that raises an OCaml exception.
However, should an error occur during MPI_Send, the handler is invoked
with the global mutex released since MPI_Send is between
enter/leave_blocking_section. Maybe that's what causing the crash.

-- 
  Olivier

Re: [Caml-list] CamlMPI: sending marshalled objects

[Jon Harrop]

On Wednesday 23 January 2008 11:52:51 Thomas Fischbacher wrote:
> What I observe (using some extra debugging code) is that doing a small
> problem where this particular send buffer is ~10MB works as expected,
> while I reproducibly get a crash in the receiver if I send a ML value
> which serializes to a 20 MB buffer. Sending suggests to be okay, though.

Note that several limits on 32-bit architectures are 16Mb.

-- 
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
http://www.ffconsultancy.com/products/?e

Re: [Caml-list] CamlMPI: sending marshalled objects

[Thomas Fischbacher]

Olivier,

>>there is an old Caml/MPI interface, written by Xavier himself in 1998.
>
> I've been using this interface some time ago (in 2006). I rewrote it a
> bit to bring it up-to-date with the current conventions.

Good to know. (I'd like to incorporate this into our own code then.)

>>As far as I can see, output_value_to_malloc() actually should be able to
>>serialize virtually any ML object (which does not contain alloc_custom()
>>blocks and similar stuff) to a buffer.
> 
> some custom blocks such as Bigarrays can be serialised too.

Of course.

>>So, (1) what is going on here, and (2) how can I repair it?
>>Is there a quick fix?
> 
> I'm not sure what is going on, I was able to transmit large marshalled
> values IIRC.

What I observe (using some extra debugging code) is that doing a small
problem where this particular send buffer is ~10MB works as expected,
while I reproducibly get a crash in the receiver if I send a ML value
which serializes to a 20 MB buffer. Sending suggests to be okay, though.

> One serious problem in the current code is the treatment of errors.
> Ocamlmpi registers an error handler that raises an OCaml exception.
> However, should an error occur during MPI_Send, the handler is invoked
> with the global mutex released since MPI_Send is between
> enter/leave_blocking_section. Maybe that's what causing the crash.

I do not think this is what gives me trouble. As I said, small problems
work, but large ones (which do the same stuff, only contain a larger
amount of data) do not.

-- 
best regards,
Thomas Fischbacher
t.fischbacher@soton.ac.uk