Re: Random segfaults / out of memory

[Goswin von Brederlow <goswin-v-b@web.de>]

Goswin von Brederlow <goswin-v-b@web.de> writes:

> Goswin von Brederlow <goswin-v-b@web.de> writes:
>
>>> On Wed, Mar 17, 2010 at 09:27:30AM +0100, Goswin von Brederlow wrote:
>>>> I want to rewrite the Digest module to expose a more lowlevel interface
>>>> to the md5 digest and add support to digest Bigarrays. I've patched the
>>>> respective files involved and it all looks alright but when I try to
>>>> build ocaml I get the following error:
>
> Ok, so I managed to bootstrap the compiler properly and build debian
> packages with my new Digest interface. But something is still wrong as I
> randomly get segfaults or
>
> <No room for growing heap
> Fatal error: out of memory.
>
> The more threads I use to compute digests in parallel the more likely
> the error becomes. But that might just be an issue with more allocations
> hapening and not a race condition between threads.

I finaly tracked down the issue with the help of Erkki Seppala.

First problem:

I had the function declared as "noalloc" but used CAMLparam2() in
it. That seems to cause random segfaults. I don't understand why but if
I remove the "noalloc" then it works.

Second problem:

When I remove the CAMLparam2() the finalizer is called too early:

CAMLprim value md5_update_bigarray(value context, value vb)
{
  //CAMLparam2(context, vb);
  struct helper *helper = (struct helper*)Data_custom_val(context);
  struct MD5Context *ctx = helper->ctx;
  fprintf(stderr, "update_bigarray: helper = %p, ctx = %p\n", helper, ctx);
  struct caml_ba_array * b = Caml_ba_array_val(vb);
  unsigned char *data = b->data;
  uintnat len = caml_ba_byte_size(b);
  caml_enter_blocking_section();
  caml_MD5Update(ctx, data, len);
  caml_leave_blocking_section();
  //CAMLreturn(Val_unit);
  return Val_unit;
}

  let rec loop () =
    Mutex.lock mutex;
    if !num = 0
    then Mutex.unlock mutex
    else begin
      decr num;
      Mutex.unlock mutex;
      let context = context () in
      let () = update_bigarray context buf
      in
	loop ()
    end
  in
    loop ()

This sometimes results in the following code flow:

 context ()				<- allocates memory
 update_bigarray context buf
   caml_enter_blocking_section();
THREAD SWITCH
 GC runs
 context is finalized			<- frees memory
THREAD SWITCH BACK
   caml_MD5Update(ctx, data, len);	<- writes to ctx which is freeed



Looks like ocamlopt really is so smart that is sees that context is
never used after the call to update_bigarray and removes it from the
root set before calling update_bigarray. It assumes the update_bigarray
will hold all its arguments alive itself, which is a valid assumption.



This is a tricky situation. The md5_update_bigarray() on its own is a
"noalloc" function. But due to the caml_enter_blocking_section() another
thread can alloc and trigger a GC run in parallel. So I guess that makes
the function actually not "noalloc".

Well, problem solved, lesson learned. :)

MfG
        Goswin