Odd behavior with GC and Mutex.lock

[Yaron Minsky <yminsky@gmail.com>]

I've been running into some very odd behavior with the GC and
Mutex.lock.  We have an application where a bunch of threads are all
pushing updates through a synchronized queue, so all of the threads are
contending for a global lock.  Under some rare circumstances, we've seen
very odd GC delays coming up, and we're at a bit of a loss to explain
them.  

The basic situation is that when the system comes under unusually high
GC load, we will sometimes see one of the threads take a long time (on
the order of half a second) when it tries to acquire the lock. We
instrumented the acquisition of the lock as follows:

let pre_stat = Gc.quick_stat () in
Mutex.lock mutex
let post_stat = Gc.quick_stat () in

The thread that gets delays shows hundreds or thousands of minor
collections and 5-10 major collections between the calls to
Gc.quick_stat ().  What's also weird is that it's always the same thread
that gets blocked, and other threads are not blocked during this period.
They seem to be able to acquire and release the lock without problems.
And the thread that does get delayed isn't even the one causing the
unusual load (although the delayed thread does do a big chunk of
allocation that is immediately unreachable right before it gets blocked).

I can't come up with a reasonable explanation for how this might happen,
and I'm hoping someone with a deeper understanding of the GC than I
might be able to help.

Yaron