Odd behavior with GC and Mutex.lock

Odd behavior with GC and Mutex.lock

[Yaron Minsky]

I've been running into some very odd behavior with the GC and
Mutex.lock.  We have an application where a bunch of threads are all
pushing updates through a synchronized queue, so all of the threads are
contending for a global lock.  Under some rare circumstances, we've seen
very odd GC delays coming up, and we're at a bit of a loss to explain
them.  

The basic situation is that when the system comes under unusually high
GC load, we will sometimes see one of the threads take a long time (on
the order of half a second) when it tries to acquire the lock. We
instrumented the acquisition of the lock as follows:

let pre_stat = Gc.quick_stat () in
Mutex.lock mutex
let post_stat = Gc.quick_stat () in

The thread that gets delays shows hundreds or thousands of minor
collections and 5-10 major collections between the calls to
Gc.quick_stat ().  What's also weird is that it's always the same thread
that gets blocked, and other threads are not blocked during this period.
They seem to be able to acquire and release the lock without problems.
And the thread that does get delayed isn't even the one causing the
unusual load (although the delayed thread does do a big chunk of
allocation that is immediately unreachable right before it gets blocked).

I can't come up with a reasonable explanation for how this might happen,
and I'm hoping someone with a deeper understanding of the GC than I
might be able to help.

Yaron

Re: Odd behavior with GC and Mutex.lock (resolved)

[Yaron Minsky]

For anyone who is interested, we've resolved this issue.  We were
mistaken to think this was a GC issue at all.  The long delays simply
resulted from some odd decisions by the Linux scheduler.  We resolved
them by changing SCHED_RR scheduling for the threads in question.

Yaron

On 8/31/05, Yaron Minsky <yminsky@gmail.com> wrote:
> I've been running into some very odd behavior with the GC and
> Mutex.lock.  We have an application where a bunch of threads are all
> pushing updates through a synchronized queue, so all of the threads are
> contending for a global lock.  Under some rare circumstances, we've seen
> very odd GC delays coming up, and we're at a bit of a loss to explain
> them.
> 
> The basic situation is that when the system comes under unusually high
> GC load, we will sometimes see one of the threads take a long time (on
> the order of half a second) when it tries to acquire the lock. We
> instrumented the acquisition of the lock as follows:
> 
> let pre_stat = Gc.quick_stat () in
> Mutex.lock mutex
> let post_stat = Gc.quick_stat () in
> 
> The thread that gets delays shows hundreds or thousands of minor
> collections and 5-10 major collections between the calls to
> Gc.quick_stat ().  What's also weird is that it's always the same thread
> that gets blocked, and other threads are not blocked during this period.
> They seem to be able to acquire and release the lock without problems.
> And the thread that does get delayed isn't even the one causing the
> unusual load (although the delayed thread does do a big chunk of
> allocation that is immediately unreachable right before it gets blocked).
> 
> I can't come up with a reasonable explanation for how this might happen,
> and I'm hoping someone with a deeper understanding of the GC than I
> might be able to help.
> 
> Yaron
>