Marshaling of custom blocks and threads

Marshaling of custom blocks and threads

[Jocelyn Serot]

Hello,

Can s/o please confirm whether the module Marshal (and especially its
usage to serialize custom blocks) is thread-safe in Ocaml 3.0 ?
I remember reading sth about a pb with bytecode threads and marshaling
in a previous version of Ocaml but cannot determine surely whether
the pb has been fixed since.

The reason i ask follows :  i have a program with (bytecode) threads
exchanging data (float arrays)  with Marshal.to_string/ThreadUnix.write and
ThreadUnix.read/Marshal.from_string.
This program works ok for small data sets (typically less than 100
floats) but crashes when the array size increases.

Cheers,

Jocelyn

-- 
E-mail: Jocelyn.Serot@l_a_s_m_e_a.u_n_i_v-bpclermont.fr .....................
S-mail: LASMEA - UMR 6602 CNRS, Universite Blaise Pascal, 63177 Aubiere cedex
Tel: (33) 04 73.40.73.30 - Fax: (33) 04 73.40.72.62 .........................
.... http://wwwlasmea.univ-bpclermont.fr/Personnel/Jocelyn.Serot/Welcome.html
Valid e-mail: remove underscores (sorry, this is prevention against junk mail

Re: Marshaling of custom blocks and threads

[Ken Wakita]

I think the Marshal module is thread unsafe because it uses a shared
buffer to produce the external image of the ML objects.

Ken Wakita

Re: Marshaling of custom blocks and threads

[Gerd Stolpmann]

On Tue, 28 Nov 2000, Ken Wakita wrote:
>I think the Marshal module is thread unsafe because it uses a shared
>buffer to produce the external image of the ML objects.

In 2.04, this was definitely true
(see http://caml.inria.fr/bin/caml-bugs/fixed?id=24;page=1;user=guest
and http://caml.inria.fr/bin/caml-bugs/fixed?id=25page=1;user=guest).
However, the bug was still subtle; the shared buffers were relatively 
well protected by the so-called "master lock". Until now, I thought
that 3.00 fixed the bug finally; i.e. the master lock really locked
the shared buffer. (I checked the code.)

Jocelyn Serot reports that Marshal is not thread-safe for custom blocks.
These did not exist in 2.04, so I suppose that only the new parts of the
marshalling system do not work properly enough. However, multi-threading
programming is error-prone, so it would be helpful to have a small program that
demonstrates the bug (normally it is sufficient to repeat the errorneous piece
of code often enough (e.g. 10000 times) to reproduce the incorrect behaviour).

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 100             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------

RE: Marshaling of custom blocks and threads

[Mattias Waldau]

The easiest way to find threading bugs are to use a multiprocessor machine.

Threading errors occurs a million times more often on a two-processor
machine than on an ordinary.

Why? Modern OS:s actually abort a thread rather seldom, and each thread is
allowed to run thousands of operations until task-switch. The chance that
this task-switch will occur at the wrong location is very small.

/mattias

-----Original Message-----
From: Pierre.Weis@inria.fr [mailto:Pierre.Weis@inria.fr]On Behalf Of
Gerd Stolpmann
Sent: Wednesday, November 29, 2000 2:22 AM
To: Ken Wakita; jserot@lasmea.univ-bpclermont.fr
Cc: caml-list@inria.fr
Subject: Re: Marshaling of custom blocks and threads


On Tue, 28 Nov 2000, Ken Wakita wrote:
>I think the Marshal module is thread unsafe because it uses a shared
>buffer to produce the external image of the ML objects.

In 2.04, this was definitely true
(see http://caml.inria.fr/bin/caml-bugs/fixed?id=24;page=1;user=guest
and http://caml.inria.fr/bin/caml-bugs/fixed?id=25page=1;user=guest).
However, the bug was still subtle; the shared buffers were relatively
well protected by the so-called "master lock". Until now, I thought
that 3.00 fixed the bug finally; i.e. the master lock really locked
the shared buffer. (I checked the code.)

Jocelyn Serot reports that Marshal is not thread-safe for custom blocks.
These did not exist in 2.04, so I suppose that only the new parts of the
marshalling system do not work properly enough. However, multi-threading
programming is error-prone, so it would be helpful to have a small program
that
demonstrates the bug (normally it is sufficient to repeat the errorneous
piece
of code often enough (e.g. 10000 times) to reproduce the incorrect
behaviour).

Gerd
--
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 100
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany
----------------------------------------------------------------------------

Re: Marshaling of custom blocks and threads

[Xavier Leroy]

> Can s/o please confirm whether the module Marshal (and especially its
> usage to serialize custom blocks) is thread-safe in Ocaml 3.0 ?
> I remember reading sth about a pb with bytecode threads and marshaling
> in a previous version of Ocaml but cannot determine surely whether
> the pb has been fixed since.

With bytecode threads, there is an issue with Marshal.{to,from}_channel,
which can raise a spurious Sys_blocked_io exception.  This problem is
still in 3.00, and will be corrected in the next release.

With system threads, as Gerd Stolpmann said, there used to be a race
condition bug in Marshal.to_string, but it is fixed in 3.00.

Finally, there is a bug in marshalling of custom objects in 3.00.
Manuel Fahndrich posted a fix, which is now in the working sources.

> The reason i ask follows :  i have a program with (bytecode) threads
> exchanging data (float arrays)  with Marshal.to_string/ThreadUnix.write and
> ThreadUnix.read/Marshal.from_string.

Shoud be OK if you apply Manuel's patch.  If it still crashes, I'd be
interested in a test program that reproduces the crash.

In a previous mail, you asked:

> Just to know - before i start the work : is there any plans or work 
> undergoing for integrating the support for Ocaml3.0 "custom blocks" in 
> the Camlidl stub code generator ?

I'm not currently working on this, and actually don't have precise
ideas about how to provide support for custom blocks.  If you have
few custom block types to handle, you can always use a typedef with
the "abstract", "c2ml" and "ml2c" attributes pointing to your own
packing/unpacking functions.

Regards,

- Xavier Leroy

RE: Marshaling of custom blocks and threads

[Manuel Fahndrich]

Here's another point to consider.

I just submitted a bug report concerning serialization of custom blocks (I
assume marshalling uses the serialization code). The deserialization code
for custom blocks does not correctly reproduce sharing. Try writing a tuple
of the same int64 data structure and reading it back via output_value,
input_value. The second component of the built tuple is garbage.

I submitted a fix which works for me:

Add:

At line 266 in file byterun/intern.c, insert the following line:

        if (intern_obj_table != NULL) intern_obj_table[obj_counter++] = v;


There's a int64 related bug in byterun/ints.c.

Change line:282 of byterun/ints.c
  <
  *wsize_64 = *wsize_64 = 8;
  >
  *wsize_32 = *wsize_64 = 8;

This bug caused the size of the block to be allocated when reading int64
data back in to be wrong on 32bit architectures.

-Manuel


-----Original Message-----
From: Gerd Stolpmann [mailto:gerd@gerd-stolpmann.de]
Sent: Tuesday, November 28, 2000 5:22 PM
To: Ken Wakita; jserot@lasmea.univ-bpclermont.fr
Cc: caml-list@inria.fr
Subject: Re: Marshaling of custom blocks and threads


On Tue, 28 Nov 2000, Ken Wakita wrote:
>I think the Marshal module is thread unsafe because it uses a shared
>buffer to produce the external image of the ML objects.

In 2.04, this was definitely true
(see http://caml.inria.fr/bin/caml-bugs/fixed?id=24;page=1;user=guest
and http://caml.inria.fr/bin/caml-bugs/fixed?id=25page=1;user=guest).
However, the bug was still subtle; the shared buffers were relatively 
well protected by the so-called "master lock". Until now, I thought
that 3.00 fixed the bug finally; i.e. the master lock really locked
the shared buffer. (I checked the code.)

Jocelyn Serot reports that Marshal is not thread-safe for custom blocks.
These did not exist in 2.04, so I suppose that only the new parts of the
marshalling system do not work properly enough. However, multi-threading
programming is error-prone, so it would be helpful to have a small program
that
demonstrates the bug (normally it is sufficient to repeat the errorneous
piece
of code often enough (e.g. 10000 times) to reproduce the incorrect
behaviour).

Gerd
-- 
----------------------------------------------------------------------------
Gerd Stolpmann      Telefon: +49 6151 997705 (privat)
Viktoriastr. 100             
64293 Darmstadt     EMail:   gerd@gerd-stolpmann.de
Germany                     
----------------------------------------------------------------------------