Re: [9fans] sleep(), sched() and ilock()

Re: [9fans] sleep(), sched() and ilock()

[Jakub Jermar]

There is a misunderstanding that I probably introduced to this thread. I was
asking two separate things: (i) how's it with ilock() and (ii) how's it with
sleep(). I certainly didn't want to put any relation between them. Sorry for
getting you confused. Now I see that my questions should have been asked
separately or at least better separated. You have answered both of the
questions in your first reply.

If you want to know where does this confusion comes from, it is from my
private kernel notes I am taking. A time ago I found out that ilock() and
iunlock()  run on the same cpu. Yesterday I reread the
notes again and I could not decide whether this fact is the goal of ilock()
or just a side effect.

The second question came up from the same notes, but had completely
different subject. It talked about sleep(), because I couldn't take easy the
fact that splhi can save the state on one cpu and splx it on another. But as
I've already replied, I perceived that that doesn't matter.

> The splx is necessary because ...... we need to return in the
> same state we arrived.

This is my point. Do we need to have the cpu in the same state as it
appeared before sleep() or does this sentence relate to processes executing
sleep()?
After your first reply and further glancing through sleep(), I would say the
latter is true.

Thanks for your patience,
Jakub Jermar

Re: [9fans] sleep(), sched() and ilock()

[presotto]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

I can't figure out what you're asking.  ilock() keeps interrupts from happening.
Therefore, unless you explicitly call sched() in the ilock'd code, you can't
move to a new processor because nothing else can run to switch the context.
Therefore, the ilock() will complete on the same processor.

If you do call sched() you're making a mistake.  You will give up the processor
with the lock still held and any interrupt that needs the lock will deadlock.
That's the reason ilock exists as opposed to lock.

The same is true of the splhi and the lock's around sleep.  The splhi is
there because wakeup can be called from interrupt level and hence the
process can't give up the processor until the locks are released.  The
splx is necessary because we may not sleep and we need to return in the
same state we arrived.  An spllo there would probably be good enough since
we don't expect sleep to be called splhi except by accident.

[-- Attachment #2: Type: message/rfc822, Size: 5177 bytes --]

From: "Jakub Jermar" <jj@comberg.cz>
To: <9fans@cse.psu.edu>
Subject: Re: [9fans] sleep(), sched() and ilock()
Date: Wed, 20 Sep 2000 06:35:33 +0200
Message-ID: <001901c022bc$58778dc0$356214d4@cz99.cz>

> First. Am I right when supposing that the reason ilock() doesn't call
> scheduler is that it has to be sure
> that iunlock() executes on the same cpu?
>
> In general, if you are doing an ilock, it should be for something that
> can be done quickly.  ilock is used when you're locking something that
> an interrupt routine also uses and scheding in the middle of it would be
> deadly.

I understand that there can be a deadlock if both a process and an interrupt
race for one lock - therefore the concept of ilock(). I also understand why
ilock() dies when it finds the ilock locked on a uni-processor - because no
one can unlock it and, seen from the other side, ilock() must have been
called twice without first calling iunlock().
But I still don't know, whether iunlock() executing on the same cpu is a
goal (maybe because of the willingness to restore the processor state as it
appeared before ilock()???) or a mere consequence of the fact that there was
no rescheduling in between.

> Second. Why doesn't sleep() simply call sched() right after it releases
both
> locks? Am I unaware of some possible race condition?
>
> Off hand, I can't see any particular reason other than tunnel vision
> on our part.  Looks like all we're doing is duplicating code with the
> net effect of saving 2 subroutine calls (splhi and sched).  I may try
> it the other way and have Gerard Holtzman verify it again and see if
> I'm missing something myself.

That's interesting. Sleep is an example of routine that goes to scheduler
splhi()'ed (as you refer to in your comment to my third point). Furthemore,
it emulates the behaviour of normal sched() after the process resumes its
p->sched: it spllo()'s but immediately after it, there is splx(x) which
cancels the effect of spllo(), no matter what x was.
Actually, this function makes me think there must be a reason for it to
duplicate piece of sched(), but I can't figure out what it is. I thought it
was because of the two locks being held, but their releasing before
gotolabel(&m->sched) is as neccessary as it would have been if sleep() had
called the normal sched().

> Third. It seems to me that sleep() might disable interrupts on a different
> cpu than is the one that enables them again before sleep() finishes. Is it
> ok when a process brings the old processor state to the new current
> processor and leaves the old one in (say) the splhi()'d state?
>
> Once you go frolicking through 'gotolabel(&m->sched)', the spl level
> is lost, i.e., schedinit always starts hi and returns low: any
> process coming out of a rescheduling comes out spllo().  That
> means that calling sched() when you are splhi() is wrong.  This
> refers back to your first point.  The processor that you did the sleep
> on will go spllo while its looking for a new process to run and will
> start that process up spllo.

But when I gotolabel(&up->sched) in sched(), after it runproc()'s a new
process, it restores its PC
and SP. SP points to the process's stack which contains the process's x
(variable from sleep()) with previous processor state, which is restored in
sleep() on the current cpu afterwards.
I believe this is the way that sleeping and waking processes carry processor
state with them. Never mind if the two processors are not the same - I
realized the functionality must be the same - it works just as though there
was no processor switch (the emphasis must be on the code and not the
processor). If the process just gets rescheduled, it leaves sched()
spllo()'ed - just as you said..

Jakub Jermar

Re: [9fans] sleep(), sched() and ilock()

[Boyd Roberts]

multi-processor or not, a quick squiz at the lyons'
commentary would explain a lot.

you can derive a lot from first principles...

Re: [9fans] sleep(), sched() and ilock()

[Russ Cox]

You never do anything that would schedule
or sleep when you hold an ilock.
Second, the ilock sets the processor splhi,
so you won't get interrupted. 

Thus iunlock always happens on the ilock
processor.

Russ

Re: [9fans] sleep(), sched() and ilock()

[presotto]

[-- Attachment #1: Type: text/plain, Size: 1560 bytes --]

	First. Am I right when supposing that the reason ilock() doesn't call
	scheduler is that it has to be sure
	that iunlock() executes on the same cpu?

In general, if you are doing an ilock, it should be for something that
can be done quickly.  ilock is used when you're locking something that
an interrupt routine also uses and scheding in the middle of it would be
deadly.

	Second. Why doesn't sleep() simply call sched() right after it releases both
	locks? Am I unaware of some possible race condition?

Off hand, I can't see any particular reason other than tunnel vision
on our part.  Looks like all we're doing is duplicating code with the
net effect of saving 2 subroutine calls (splhi and sched).  I may try
it the other way and have Gerard Holtzman verify it again and see if
I'm missing something myself.

	Third. It seems to me that sleep() might disable interrupts on a different
	cpu than is the one that enables them again before sleep() finishes. Is it
	ok when a process brings the old processor state to the new current
	processor and leaves the old one in (say) the splhi()'d state?

Once you go frolicking through 'gotolabel(&m->sched)', the spl level
is lost, i.e., schedinit always starts hi and returns low: any
process coming out of a rescheduling comes out spllo().  That
means that calling sched() when you are splhi() is wrong.  This
refers back to your first point.  The processor that you did the sleep
on will go spllo while its looking for a new process to run and will
start that process up spllo.

[-- Attachment #2: Type: message/rfc822, Size: 2225 bytes --]

From: "Jakub Jermář" <jj@comberg.cz>
To: "9fans" <9fans@cse.psu.edu>
Subject: [9fans] sleep(), sched() and ilock()
Date: Wed, 20 Sep 2000 04:40:12 +0200
Message-ID: <000501c022ac$54713b00$9a6214d4@cz99.cz>

I am now reconsidering some aspects of sleeping, scheduling and ilocking
that I considered clear.

First. Am I right when supposing that the reason ilock() doesn't call
scheduler is that it has to be sure
that iunlock() executes on the same cpu?

Second. Why doesn't sleep() simply call sched() right after it releases both
locks? Am I unaware of some possible race condition?

Third. It seems to me that sleep() might disable interrupts on a different
cpu than is the one that enables them again before sleep() finishes. Is it
ok when a process brings the old processor state to the new current
processor and leaves the old one in (say) the splhi()'d state?

Jakub Jermar

Re: [9fans] sleep(), sched() and ilock()

[Jakub Jermar]

> multi-processor or not, a quick squiz at the lyons'
> commentary would explain a lot.
>
> you can derive a lot from first principles...
>

I am not familiar with the lyons's commentary (several days ago, I read some
bad criticism on it in 9fans), but I am pretty familiar with the principles
of interrupt and lock protecting.

I exactly know what the code is doing and what is it used for, but I can't
distinguish between the goals and the consequences (or -if you want- side
effects). Again, I don't ask about the functionality, I just want to know
what is the goal and what is the consequence.

As for the multi- versus uni-processors: there are important issues that
make the behaviour different.
As an evidence of this, you can find uni-processor branches of code that try
to avoid deadlock, which simply cannot happen on SMP (in the same
situation). A good example of what I've just said is ilock() itself.

Jakub Jermar

Re: [9fans] sleep(), sched() and ilock()

[Jakub Jermar]

> First. Am I right when supposing that the reason ilock() doesn't call
> scheduler is that it has to be sure
> that iunlock() executes on the same cpu?
>
> In general, if you are doing an ilock, it should be for something that
> can be done quickly.  ilock is used when you're locking something that
> an interrupt routine also uses and scheding in the middle of it would be
> deadly.

I understand that there can be a deadlock if both a process and an interrupt
race for one lock - therefore the concept of ilock(). I also understand why
ilock() dies when it finds the ilock locked on a uni-processor - because no
one can unlock it and, seen from the other side, ilock() must have been
called twice without first calling iunlock().
But I still don't know, whether iunlock() executing on the same cpu is a
goal (maybe because of the willingness to restore the processor state as it
appeared before ilock()???) or a mere consequence of the fact that there was
no rescheduling in between.

> Second. Why doesn't sleep() simply call sched() right after it releases
both
> locks? Am I unaware of some possible race condition?
>
> Off hand, I can't see any particular reason other than tunnel vision
> on our part.  Looks like all we're doing is duplicating code with the
> net effect of saving 2 subroutine calls (splhi and sched).  I may try
> it the other way and have Gerard Holtzman verify it again and see if
> I'm missing something myself.

That's interesting. Sleep is an example of routine that goes to scheduler
splhi()'ed (as you refer to in your comment to my third point). Furthemore,
it emulates the behaviour of normal sched() after the process resumes its
p->sched: it spllo()'s but immediately after it, there is splx(x) which
cancels the effect of spllo(), no matter what x was.
Actually, this function makes me think there must be a reason for it to
duplicate piece of sched(), but I can't figure out what it is. I thought it
was because of the two locks being held, but their releasing before
gotolabel(&m->sched) is as neccessary as it would have been if sleep() had
called the normal sched().

> Third. It seems to me that sleep() might disable interrupts on a different
> cpu than is the one that enables them again before sleep() finishes. Is it
> ok when a process brings the old processor state to the new current
> processor and leaves the old one in (say) the splhi()'d state?
>
> Once you go frolicking through 'gotolabel(&m->sched)', the spl level
> is lost, i.e., schedinit always starts hi and returns low: any
> process coming out of a rescheduling comes out spllo().  That
> means that calling sched() when you are splhi() is wrong.  This
> refers back to your first point.  The processor that you did the sleep
> on will go spllo while its looking for a new process to run and will
> start that process up spllo.

But when I gotolabel(&up->sched) in sched(), after it runproc()'s a new
process, it restores its PC
and SP. SP points to the process's stack which contains the process's x
(variable from sleep()) with previous processor state, which is restored in
sleep() on the current cpu afterwards.
I believe this is the way that sleeping and waking processes carry processor
state with them. Never mind if the two processors are not the same - I
realized the functionality must be the same - it works just as though there
was no processor switch (the emphasis must be on the code and not the
processor). If the process just gets rescheduled, it leaves sched()
spllo()'ed - just as you said..

Jakub Jermar

[9fans] sleep(), sched() and ilock()

[Jakub Jermář]

I am now reconsidering some aspects of sleeping, scheduling and ilocking
that I considered clear.

First. Am I right when supposing that the reason ilock() doesn't call
scheduler is that it has to be sure
that iunlock() executes on the same cpu?

Second. Why doesn't sleep() simply call sched() right after it releases both
locks? Am I unaware of some possible race condition?

Third. It seems to me that sleep() might disable interrupts on a different
cpu than is the one that enables them again before sleep() finishes. Is it
ok when a process brings the old processor state to the new current
processor and leaves the old one in (say) the splhi()'d state?

Jakub Jermar