From mboxrd@z Thu Jan  1 00:00:00 1970
From: erik quanstrom <quanstro@quanstro.net>
Date: Thu, 19 Jun 2014 12:01:10 -0400
To: 9fans@9fans.net
Message-ID: <399b6c5c08b7aeab53f52ce0414714bf@brasstown.quanstro.net>
MIME-Version: 1.0
Content-Type: text/plain; charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Subject: [9fans] cache lines, and 60000 cycles of doom
Topicbox-Message-UUID: fb7a1034-ead8-11e9-9d60-3106f5b1d025

i'm seeing some mighty interesting timing on my intel ivy bridge.
i found a bug in the file server aoe implementation (can't happen
if you're using the uniprocessor x86 version) that happens because
the Srb is freed before wakeup completes.  to solve this there is
some code that sets the state (this is from ken's ancient scheduler,
by way of sape)

	wakeup(&srb);
	srb->state = Free;

code that receives it is like this

	sleep(&srb, srbdone, srb);
	cycles(&t0);
	for(n = 0; srb->state != Free; n++){
		if(srb->wmach == m->machno)
			sched();
		else
			monmwait(&srb->state, Alloc);
	}
	cycles(&t1);
	free(srb);

the astounding thing is that t1-t0 is often ~ 60,000 cycles.
it only hits a small fraction of the time, and the average is
much lower.  but that just blows the mind.  60000 cycles!

(other versions with sched were much worse.)

as far as i can tell, there are no funny bits in the scheduler that
can cause this, and no wierd scheduling is going on.

i'm baffled.

- erik