Re: [9fans] Kernel question: i386 test-and-set problem

[miller@hamnavoe.demon.co.uk]

presotto@plan9.bell-labs.com writes:
> If I have two locks,
> one in r and one in p, I can't figure out how to get them both to lock
> in the same order.

My suggestion is below.  The interaction between sleep() and wakeup() is
exactly as in the beautiful and succinct algorithm in your paper, using
the lock on r.  Adding postnote() introduces the pointer p->r to enable
postnote() to find r, so the lock p->rlock is introduced to protect it
in the interaction between sleep() and postnote().  The interaction
between wakeup() and postnote() is protected by the lock on r.

Eliminating interrupt dis/en-abling for simplicity:

sleep(r):    {'up' is the current Proc}
	lock(up->rlock)
	lock(r)
	if wakeup condition is satisfied or up->notepending != 0
		unlock(r)
		unlock(up->rlock)
	else
		r->p = up
		up->r = r
		up->state = Wakeme
		unlock(r)
		unlock(up->rlock)
		-- reschedule --
	if up->notepending
		up->notepending = 0
		error(Eintr)

wakeup(r):    {wakeup condition is satisfied}
	lock(r)
	p = r->p
	if p != 0
		r->p = 0
		p->r = 0	
		ready(p)
	unlock(r)

postnote(p):
	p->notepending = 1
	lock(p->rlock)
	r = p->r
	if r != 0
		lock(r)
		if(r->p == p)
			r->p = 0
			p->r = 0
			ready(p)
		unlock(r)
	unlock(p->rlock)

Explanation: we can think of locks in terms of the invariants they
protect.  Define S(p,r) = "Proc p is sleeping or committed to
sleeping on Rendez r", where committed means that p will sleep on r
without first testing the wakeup condition or testing for pending
notes.  Define T(p,r) =  "Proc p is not sleeping on r, and will not
sleep on r without first testing both the wakeup condition and
notepending".  The significance of these predicates is that
if S(p,r) holds, then wakeup(r) or postnote(p) must take steps
to awaken p from r; whereas if T(p,r) holds, then wakeup(r) or
postnote(p) need not -- must not -- do so.

The invariant we want to maintain on each Rendez r is:

IR(r): { for all p: (p = r->p) <=> S(p,r) }

or informally, r->p points to a Proc if and only if that Proc is
sleeping or committed to sleeping on r.  Any critical section of
code where IR(r) is temporarily untrue (e.g. while testing conditions
or manipulating r->p) must be protected by locking r.  Because the
inverse of S(p,r) is not quite the same as T(p,r) -- p may have
tested only one of the two conditions -- we also impose a global
invariant:

IPR: { for all p,r: !S(p,r) => T(p,r) }

To protect this invariant, testing of the wakeup condition and
notepending before sleeping on r must be within a lock on r.
The invariant for Proc p, protected by p->rlock, is weaker than IR:

IP(p): { for all r: S(p,r) => (r = p->r) }

or informally, if p is sleeping or committed to sleeping on a Rendez,
then p->r points to that Rendez.  Note that the reverse implication
is not stipulated: if p->r points to a Rendez, we can't necessarily
conclude that S(p,p->r) holds; only that !S(p,r) -- and therefore
T(p,r), by IPR -- holds for all other r.  But the uncertainty over
S(p,p->r) can be resolved using IR(p->r).

Key point: wakeup() can safely set p->r = 0 without locking p->rlock,
because this assignment does not affect the invariant IP(p).  In fact
the assignment could be deleted and the algorithm would still work;
it is only an optimisation to avoid postnote() looking at a Rendez
unnecessarily.  This observation is what saves us from the dilemma
of locking order.

-- Richard Miller