Re: [9fans] cache lines, and 60000 cycles of doom

[Bakul Shah <bakul@bitblocks.com>]

On Thu, 19 Jun 2014 12:01:10 EDT erik quanstrom <quanstro@quanstro.net> wrote:
> i'm seeing some mighty interesting timing on my intel ivy bridge.
> i found a bug in the file server aoe implementation (can't happen
> if you're using the uniprocessor x86 version) that happens because
> the Srb is freed before wakeup completes.  to solve this there is
> some code that sets the state (this is from ken's ancient scheduler,
> by way of sape)
>
> 	wakeup(&srb);
> 	srb->state = Free;
>
> code that receives it is like this
>
> 	sleep(&srb, srbdone, srb);
> 	cycles(&t0);
> 	for(n = 0; srb->state != Free; n++){
> 		if(srb->wmach == m->machno)
> 			sched();
> 		else
> 			monmwait(&srb->state, Alloc);
> 	}
> 	cycles(&t1);
> 	free(srb);
>
> the astounding thing is that t1-t0 is often ~ 60,000 cycles.
> it only hits a small fraction of the time, and the average is
> much lower.  but that just blows the mind.  60000 cycles!
>
> (other versions with sched were much worse.)
>
> as far as i can tell, there are no funny bits in the scheduler that
> can cause this, and no wierd scheduling is going on.
>
> i'm baffled.

Could there've been a context switch?