[9fans] why not halt in x86 multicore

[9fans] why not halt in x86 multicore

[Henning Schild]

Hi,

in pc/main.c idlehands() the kernel decides to release the CPU only
when you are not running a multicore kernel. As a result there is only
busy waiting. Unfortunately there is no hint telling why we do not let
the CPU rest a little.

> void
> idlehands(void)
> {
> 	if(conf.nmach == 1)
> 		halt();
> }

I am running a mostly idle plan9 VM on my laptop and it causes the
battery to discharge in no time. Right now my kernel has the condition
commented out and it seems to work fine. Still i would like
to understand why this condition was put there. It is probably causing a
lot of energy to be wasted which should better be for a good reason.

Henning

Re: [9fans] why not halt in x86 multicore

[erik quanstrom]

On Fri Jul  1 13:06:36 EDT 2011, henning@plan9.bell-labs.com wrote:
> in pc/main.c idlehands() the kernel decides to release the CPU only
> when you are not running a multicore kernel. As a result there is only
> busy waiting. Unfortunately there is no hint telling why we do not let
> the CPU rest a little.
>
> > void
> > idlehands(void)
> > {
> > 	if(conf.nmach == 1)
> > 		halt();
> > }
>
> I am running a mostly idle plan9 VM on my laptop and it causes the
> battery to discharge in no time. Right now my kernel has the condition
> commented out and it seems to work fine. Still i would like
> to understand why this condition was put there. It is probably causing a
> lot of energy to be wasted which should better be for a good reason.

right!  this is an interesting problem.

suppose that i have a multicore machine and all maches
have executed halt().  then, if i get an interrupt on one that
causes a proc to be ready, then i have to wait until the next
clock tick to sched() it on any other core.  this is going to be a
huge performance hit, especially if you are using a 100hz clock
as the distribution does.  for me a potential latency on the order
of 1-10ms is not going to cut it.

there are a number of potential solutions to this that i'd like to
try, but i haven't had any time yet.

- erik

Re: [9fans] why not halt in x86 multicore

[Ali Mashtizadeh]

The hlt instruction on x86 leaves the halt state after an external
interrupt is fired. I'm not sure why you think it doesn't work? If you
need to wake up other cores then you may need IPIs is that what you
mean?

~ Ali

On Fri, Jul 1, 2011 at 10:23 AM, erik quanstrom <quanstro@quanstro.net> wrote:
> On Fri Jul  1 13:06:36 EDT 2011, henning@plan9.bell-labs.com wrote:
>> in pc/main.c idlehands() the kernel decides to release the CPU only
>> when you are not running a multicore kernel. As a result there is only
>> busy waiting. Unfortunately there is no hint telling why we do not let
>> the CPU rest a little.
>>
>> > void
>> > idlehands(void)
>> > {
>> >     if(conf.nmach == 1)
>> >             halt();
>> > }
>>
>> I am running a mostly idle plan9 VM on my laptop and it causes the
>> battery to discharge in no time. Right now my kernel has the condition
>> commented out and it seems to work fine. Still i would like
>> to understand why this condition was put there. It is probably causing a
>> lot of energy to be wasted which should better be for a good reason.
>
> right!  this is an interesting problem.
>
> suppose that i have a multicore machine and all maches
> have executed halt().  then, if i get an interrupt on one that
> causes a proc to be ready, then i have to wait until the next
> clock tick to sched() it on any other core.  this is going to be a
> huge performance hit, especially if you are using a 100hz clock
> as the distribution does.  for me a potential latency on the order
> of 1-10ms is not going to cut it.
>
> there are a number of potential solutions to this that i'd like to
> try, but i haven't had any time yet.
>
> - erik
>
>

Re: [9fans] why not halt in x86 multicore

[Russ Cox]

On Fri, Jul 1, 2011 at 1:47 PM, Ali Mashtizadeh <mashtizadeh@gmail.com> wrote:
> The hlt instruction on x86 leaves the halt state after an external
> interrupt is fired. I'm not sure why you think it doesn't work?

he didn't say it didn't work.
he said it required waiting for the next interrupt,
of which the only guaranteed one is the next
clock interrupt, which might be a long time coming.

interprocessor interrupts are the obvious solution
but no one has bothered to implement them.

Re: [9fans] why not halt in x86 multicore

[erik quanstrom]

>
> interprocessor interrupts are the obvious solution
> but no one has bothered to implement them.
>

i almost mentioned this, and it's on my list.  i just haven't
had the time.  it turns out that a naive implementation of
ipi'ing 1 proc per wakeup will have to watch out for the
same races that bedeviled multiprocessor sleep/wakeup.

the really naive solution of waking 'em all up is a very
bad idea because you'll have n-1 procs duking it out
over the run queue.

perhaps i just ignorant of the standard solution.
or perhaps the sleep/wakeup model isn't the right one
for the situation.

- erik

Re: [9fans] why not halt in x86 multicore

[Bakul Shah]

On Fri, 01 Jul 2011 14:01:28 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> >
> > interprocessor interrupts are the obvious solution
> > but no one has bothered to implement them.
> >
>
> i almost mentioned this, and it's on my list.  i just haven't
> had the time.  it turns out that a naive implementation of
> ipi'ing 1 proc per wakeup will have to watch out for the
> same races that bedeviled multiprocessor sleep/wakeup.
>
> the really naive solution of waking 'em all up is a very
> bad idea because you'll have n-1 procs duking it out
> over the run queue.
>
> perhaps i just ignorant of the standard solution.
> or perhaps the sleep/wakeup model isn't the right one
> for the situation.

You might want to look at how FreeBSD does.
    http://fxr.watson.org/fxr/source/i386/i386/mp_machdep.c
    http://fxr.watson.org/fxr/source/i386/i386/machdep.c
Robert Watson's excellent site has cross-referenced sources
for a variety of OSes (including plan9) and releases for some
of them.

FreeBSD put in hlt for MP kernels in 2003 but the code has
evolved since then. The commit logs may be of some use too.
See the same files in http://www.freebsd.org/cgi/cvsweb.cgi

Re: [9fans] why not halt in x86 multicore

[Venkatesh Srinivas]

On i386 systems with MONITOR/MWAIT, you could use them; simpler than
adding support for IPIs, I imagine.

-- vs

Re: [9fans] why not halt in x86 multicore

[Henning Schild]

On Fri, 1 Jul 2011 19:23:03 +0200
erik quanstrom <quanstro@quanstro.net> wrote:

> suppose that i have a multicore machine and all maches
> have executed halt().  then, if i get an interrupt on one that
> causes a proc to be ready, then i have to wait until the next
> clock tick to sched() it on any other core.  this is going to be a
> huge performance hit, especially if you are using a 100hz clock
> as the distribution does.  for me a potential latency on the order
> of 1-10ms is not going to cut it.
>
> there are a number of potential solutions to this that i'd like to
> try, but i haven't had any time yet.

I am aware of the latency issue there but in my case i care much more
about the battery of my host than about responsiveness of the guest.

So far letting it halt() works fine for me. I guess i could run into
some fun caused by missing TLB invalidation if i used that kernel on a
"real" machine. As far as i know plan9 signals the need for TLB
invalidation through a datastructure. Assuming that everybody polls
with their clock frequency.
So i guess to get it right i will have to clear the TLBs after a
halt(). Right?

Henning

Re: [9fans] why not halt in x86 multicore

[Sape Mullender]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

Indeed, there's an assumption that each core manages the TLB by
examining the data structures at clock frequency.  In a halt,
the clock interrupts still happen so I guess there's no need to
invalidate the TLBs just before halt.

You could set a variable indicating you're halted and cause other
cores to send an interprocessor interrupt when a process gets
activated.  On the other hand, the non-halted cores will also
run the scheduler and pick up any recently readied process.

	Sape

[-- Attachment #2: Type: message/rfc822, Size: 3207 bytes --]

From: Sape Mullender <sape@plan9.bell-labs.com>
To: "9fans@9fans.net" <9fans@9fans.net>
Subject: Re: [9fans] why not halt in x86 multicore
Date: Mon, 4 Jul 2011 11:24:20 +0200
Message-ID: <20110704112420.16237b22@leffe.cs.bell-labs.com>

On Fri, 1 Jul 2011 19:23:03 +0200
erik quanstrom <quanstro@quanstro.net> wrote:

> suppose that i have a multicore machine and all maches
> have executed halt().  then, if i get an interrupt on one that
> causes a proc to be ready, then i have to wait until the next
> clock tick to sched() it on any other core.  this is going to be a
> huge performance hit, especially if you are using a 100hz clock
> as the distribution does.  for me a potential latency on the order
> of 1-10ms is not going to cut it.
>
> there are a number of potential solutions to this that i'd like to
> try, but i haven't had any time yet.

I am aware of the latency issue there but in my case i care much more
about the battery of my host than about responsiveness of the guest.

So far letting it halt() works fine for me. I guess i could run into
some fun caused by missing TLB invalidation if i used that kernel on a
"real" machine. As far as i know plan9 signals the need for TLB
invalidation through a datastructure. Assuming that everybody polls
with their clock frequency.
So i guess to get it right i will have to clear the TLBs after a
halt(). Right?

Henning

Re: [9fans] why not halt in x86 multicore

[erik quanstrom]

> I am aware of the latency issue there but in my case i care much more
> about the battery of my host than about responsiveness of the guest.

so why don't you just give your plan 9 guest 1 processor, either through the vmm
or *ncpu, and be done with it?

- erik

Re: [9fans] why not halt in x86 multicore

[Steve Simon]

> so why don't you just give your plan 9 guest 1 processor, either through the vmm
> or *ncpu, and be done with it?

Too much work to port plan9 a cpu server to the mac ppc hardware... :-)

An intel mac would solve many problems I agree.

-Steve

Re: [9fans] why not halt in x86 multicore

[erik quanstrom]

On Mon Jul  4 09:14:10 EDT 2011, steve@quintile.net wrote:
> > so why don't you just give your plan 9 guest 1 processor, either through the vmm
> > or *ncpu, and be done with it?
>
> Too much work to port plan9 a cpu server to the mac ppc hardware... :-)
>
> An intel mac would solve many problems I agree.

i was responding to the thread on halt() and multicore.

- erik

Re: [9fans] why not halt in x86 multicore

[Steve Simon]

I appologise for the noise, got my emails very confused.

-Steve