Random segfaults / out of memory [Was: Re: [Caml-list] Rewriting the Digest module causes linking errors]

[Goswin von Brederlow <goswin-v-b@web.de>]

Goswin von Brederlow <goswin-v-b@web.de> writes:

>> On Wed, Mar 17, 2010 at 09:27:30AM +0100, Goswin von Brederlow wrote:
>>> I want to rewrite the Digest module to expose a more lowlevel interface
>>> to the md5 digest and add support to digest Bigarrays. I've patched the
>>> respective files involved and it all looks alright but when I try to
>>> build ocaml I get the following error:

Ok, so I managed to bootstrap the compiler properly and build debian
packages with my new Digest interface. But something is still wrong as I
randomly get segfaults or

<No room for growing heap
Fatal error: out of memory.

The more threads I use to compute digests in parallel the more likely
the error becomes. But that might just be an issue with more allocations
hapening and not a race condition between threads.

Can anyone spot the bug?

I split the patch into two so it is easier to bootstrap. Apply
0008-md5-rewrite.patch, build ocamlc and copy it to boot and then apply
0009-digest-rewrite.patch and build everything. Debian user can also get
the debian source or amd64 debs.

Patches and source at: http://mrvn.homeip.net/ocaml/

Testcode:
----------------------------- foo.ml ---------------------------------
open Bigarray

let blocksize = 1048576
let num = ref 1024
let mutex = Mutex.create ()

let compute x =
  Printf.printf "Thread %d started\n" x; flush_all ();
  let buf = Array1.create int8_unsigned  c_layout blocksize in
  let rec loop () =
    Mutex.lock mutex;
    if !num = 0
    then Mutex.unlock mutex
    else begin
      decr num;
      Mutex.unlock mutex;
      ignore (Array1.Digest.bigarray buf);
      loop ()
    end
  in
    loop ()

let main x =
  Printf.printf "Running with %d threads\n" x; flush_all ();
  let rec loop acc = function
      0 -> acc
    | x ->
	let thread = Thread.create compute x
	in
	  loop (thread :: acc) (x - 1)
  in
  let threads = loop [] x
  in
    List.iter Thread.join threads;
    Printf.printf "All done\n"

let _ =
  let control = Gc.get () in
  let _ = control.Gc.verbose <- 0x3ff in
  let _ = Gc.set control
  in
    Gc.compact ();
    flush_all ();
    Scanf.sscanf Sys.argv.(1) "%d" main;
    Gc.compact ();
    flush_all ()
----------------------------------------------------------------------

ocamlopt -thread -o foo unix.cmxa threads.cmxa bigarray.cmxa foo.ml && \
time ./foo 1 && time ./foo 2 && time ./foo 3 && time ./foo 4 && \
time ./foo 16

1-4 usualy runs while 16 usualy breaks. But sometimes it breaks earlier
or 16 runs too. Just repeat ./foo 16 till it fails.

MfG
        Goswin

PS: Runtimes without the verbose GC:
./foo 1  7.28s user 0.01s system 100% cpu 7.291 total
./foo 2  8.75s user 0.02s system 196% cpu 4.474 total
./foo 3  9.76s user 0.03s system 298% cpu 3.285 total
./foo 4  10.69s user 0.04s system 371% cpu 2.891 total