[9fans] IRQs again...

[9fans] IRQs again...

[andrey mirtchovski]

We finally got around to looking at booting R4 without BIOS again
(exploring the niceties of the 4th edition took a bit longer than
expected) and have to report some problems with the way the new IRQ
handling routines are, umm, handling IRQs in a BIOS-less environment.

Here's the sequence of events as seen by /sys/src/9/pc/pci.c:

The southbridge is recognized as ALI, the get and set routines become
ali_get() and ali_set().  The pci routing table is found at 0xf0000
(LinuxBios puts it there -- a left over from our R3 work) and is
printed as:

% pcir
PCI interrupt routing table version 1.0 at f0000
South Bridge 0.7.0, irqs 0000 compat 10B9/1533
miniport data: 00 00 00 00
0.12.0 01:	[0] 01 1EB8 [1] 02 1EB8 [2] 03 1EB8 [3] 04 1EB8
0.11.0 02:	[0] 02 1EB8 [1] 03 1EB8 [2] 04 1EB8 [3] 01 1EB8
0.10.0 03:	[0] 03 1EB8 [1] 04 1EB8 [2] 01 1EB8 [3] 02 1EB8
0.9.0 04:	[0] 04 1EB8 [1] 01 1EB8 [2] 02 1EB8 [3] 03 1EB8
0.13.0 05:	[0] 04 1EB8 [1] 01 1EB8 [2] 02 1EB8 [3] 03 1EB8
0.14.0 06:	[0] 03 1EB8 [1] 04 1EB8 [2] 01 1EB8 [3] 02 1EB8
0.6.0 00:	[0] 08 1EB8 [1] 00 1EB8 [2] 00 1EB8 [3] 00 1EB8
0.20.0 00:	[0] 59 1EB8 [1] 00 1EB8 [2] 00 1EB8 [3] 00 1EB8
1.0.0 00:	[0] 01 1EB8 [1] 00 1EB8 [2] 00 1EB8 [3] 00 1EB8
%

Then the code tries to set the IRQs according to the pci routing
table, and that's where the problems start.  The ali_get() routine
seems to be very similar to the one linux uses, with several
exceptions you can see below:

	plan9:ali_get():
		pirq = pcicfgr8(router, 0x48 + ((link-1)>>1));
		return (link & 1)? map[pirq&15]: map[pirq>>4];

	linux:pirq_ali_get():
 	       return irqmap[read_config_nybble(router, 0x48, pirq-1)];

I'm not sure what warrants these changes in plan9, but as an effect
ali_get() returns '0' as the interrupt for all pci devices it is
executed for.

If changed to:

		pirq = pcicfgr8(router, 0x48 + (link-1));
		returnmap[pirq];

(which corresponds more closely to the linux code), we end up with
_some_ PCI devices having irqs as follows:

bus dev type vid  did intl memory
0   0/0 06 00 00 10b9 1621   0
0   1/0 06 04 00 10b9 5247   0  ->1
0   6/0 04 01 00 10b9 5451   6  0:00001001 256 1:81000000 4096
0   7/0 06 01 00 10b9 1533   0
0  11/0 02 00 00 10ec 8139   0  0:00001101 256 1:81001000 256
0  12/0 03 00 00 102b 0519   0  0:81004000 16384 1:84000008 8388608
0  16/0 01 01 fa 10b9 5229   0  0:000001f1 65040 1:000003f5 64528 2:00000171 65168 3:00000375 64656 4:00001201 16
0  17/0 00 00 00 10b9 7101   0
0  20/0 0c 03 10 10b9 5237   9  0:81008000 4096
1   0/0 03 00 00 10de 00a0   0  0:80000000 16777216 1:82000008 33554432

We had an almost working version of the above code written for R3. The
crucial piece we (I) were missing was setting the the IRQ in the
southbridge after obtaining it from the routing table.  It was by no
means comprehensive and was tailored to the motherboards we were
testing on.

The specification used was, ironically, obtained from microsoft's web
site: http://www.microsoft.com/hwdev/archive/BUSBIOS/pciirq.asp

The significant difference was that according to this specification
the IRQs are to be obtained from the last two bytes of the info
structure (pciinfo is the name linux and bsd use for it).  The first
byte is passed as 'link' to ali_get() in the plan9 code (m[0] is the
variable), however m[1] and m[2] form another 16 bit variable, having
1 in the place of each IRQ which is routable on the board (1eb8 for
our test motherboard).  Going through the bits in search for ones
gives us the interrupts we can use.

We're not particularly sure this is the proper way of doing things
(both linux and freebsd seem to be doing it at one stage or another of
the pci routing code, but...).

andrey

ps: then again, i may have missed something important, in which case
feel free to clue me in

Re: [9fans] IRQs again...

[Ronald G Minnich]

On Tue, 28 May 2002, David Gordon Hogan wrote:

> Do you actually have a hot-swappable cPCI system, or
> is this all hypothetical?

I've got one, plan to get more, and would like to fix this. Plus the one I
have (www.cluster-labs.de) is full of nodes that ONLY run linuxbios. You
can't run a legacy bios. So we've got to fix this for our own uses.
Otherwise everything just keeps running Linux.

I'm willing to fix this in linuxbios, in fact I started to, just for Plan
9. What stopped me cold was looking at all the corner cases -- it seems no
matter what we do we'll always be doing something wrong for some
configurations -- not even versions, configurations -- of some OSes. Ouch.
I think this is why Linux has started ignoring a lot of the BIOS settings
and just does its own.

> I think our main reason is that (we think) we've got better
> things to do.

That's a good reason. I mean that. There's always too much work. I was
worried that there was a technical gotcha.

> So the `guessing' code probably isn't used all that often.

Except in our lab ... where it is used a lot and does seem to do ok.

> But, by all means, don't let me stop you from adding hot
> swap support to the kernel :-)

Works for me. We'll see what we can do.

Thanks.

ron

Re: [9fans] IRQs again...

[David Gordon Hogan]

> No. Sorry. There are good reasons for this and it ain't going to happen.
> We've been over this ground quite a bit over the last 2 years. The OS has
> a way better idea of what it needs, and once you start pulling in nonsense
> like SMP and APIC and IO-APIC it gets very very messy. The odds that the
> BIOS can get things set up right for the OS become vanishingly small.

Yet somehow, most of the BIOSes we've seen do a pretty good
job of it.  I think it comes down to the BIOS being board-specific.
You're trying to replace such a thing with a generic program
(AFAICT), which I imagine to be much harder.  Presumably
you have some way of configuring it though...?

> I'm still a little lost on how you plan to handle the hot plug thing. What
> do you plan to do if the device is not there until after the OS boots? Or
> goes away? do you reclaim the interrupt or not? Do you plan to callback to
> the BIOS? We don't do callbacks either.

Just divvy up the free IRQs amongst the various PCI interrupt
lines on the south bridge.  When a new device is inserted,
consult the PCI routing table and the south bridge assignments
to figure out its IRQ.  Remember that PCI IRQs can be, and
are often, shared.  On many systems, most of the IRQs are
soaked up by `legacy' devices, leaving 2 or 3 for PCI.

Do you actually have a hot-swappable cPCI system, or
is this all hypothetical?

>> We've thought about what it would take to make Plan 9
>> do this instead, and generally reached the conclusion
>> that it would be a Bad Thing.
>
> can you elaborate?
>
> I just don't agree, and neither (it seems) do the other open source OSes
> out there. But I've been wrong lots of times so if you have a solid reason
> I'm all ears.

I think our main reason is that (we think) we've got better
things to do.

I've looked at Linux's approach, and it doesn't seem to be a
real solution: as there is no way to be sure of what IRQs are
in use by legacy devices, it just sort of guesses.  Most of the
time the BIOS has already allocated IRQs, if not to the
device itself, at least to the interrupt lines (as controlled by
the south bridge).  So the `guessing' code probably isn't used
all that often.

But, by all means, don't let me stop you from adding hot
swap support to the kernel :-)

Re: [9fans] IRQs again...

[Ronald G Minnich]

On Mon, 27 May 2002, David Gordon Hogan wrote:

> I would like to suggest that since LinuxBios has "BIOS" in
> the name, that it should allocate IRQs to the various devices
> as part of system initialisation, as this appears to be one of
> the (de facto) jobs of the BIOS.  Every other BIOS does it.

No. Sorry. There are good reasons for this and it ain't going to happen.
We've been over this ground quite a bit over the last 2 years. The OS has
a way better idea of what it needs, and once you start pulling in nonsense
like SMP and APIC and IO-APIC it gets very very messy. The odds that the
BIOS can get things set up right for the OS become vanishingly small.

I'm still a little lost on how you plan to handle the hot plug thing. What
do you plan to do if the device is not there until after the OS boots? Or
goes away? do you reclaim the interrupt or not? Do you plan to callback to
the BIOS? We don't do callbacks either.

Of course Andrey might force me to change my mind.

> We've thought about what it would take to make Plan 9
> do this instead, and generally reached the conclusion
> that it would be a Bad Thing.

can you elaborate?

I just don't agree, and neither (it seems) do the other open source OSes
out there. But I've been wrong lots of times so if you have a solid reason
I'm all ears.

ron

Re: [9fans] IRQs again...

[Ronald G Minnich]

On Mon, 27 May 2002 jmk@plan9.bell-labs.com wrote:
> We've been there and done that, both running the Plan 9 code base
> on hot-plug systems and using a Plan 9 kernel as the boot ROM (is it
> really almost 10 years ago?).

Everybody has. I did it 25 years ago on all kinds of stuff, with all kinds
of OSes, some Unix, some not. I don't think our past successes, however
impressive, are the issue here. Having the OS config interrupts is old
news. So we don't need (I think) to discuss how we all did it, unless the
result is a chunk of code in the same email. It's just not productive.

The issue is, that we need to do it now on stupid PC hardware, with stupid
M$-designed "$PIR" tables and stupid broken PC chipsets. My first pass
through 4e source shows there are lots of problems with how plan9 is doing
it and that's what we're trying to figure out. We've fixed one or two
things but we're still chasing all the problems. Linux gets it right, and
plan9 doesn't, and that bothers me quite a bit. I want Plan 9 to be as
good as Linux in this area. I hope that is a shared goal.

> Before you flashed LinuxBIOS on the motherboard, is it likely Plan 9
> would have booted?

Sure, on the one board it does, but that fact has not been terribly
helpful, esp. as it seems plan9 did not always configure things right even
under the BIOS -- these BIOSes do amazingly confusing things for good
OSes to get mixed up by.

ron

Re: [9fans] IRQs again...

[David Gordon Hogan]

> I forgot to mention that if we attach a R4 hard drive to the box
> with LinuxBios on then we boot fine.  The IRQs are of course not
> set correctly (i.e.  either all devices have IRQ==0 or some other
> incoherent arrangement).

I would like to suggest that since LinuxBios has "BIOS" in
the name, that it should allocate IRQs to the various devices
as part of system initialisation, as this appears to be one of
the (de facto) jobs of the BIOS.  Every other BIOS does it.

We've thought about what it would take to make Plan 9
do this instead, and generally reached the conclusion
that it would be a Bad Thing.

Re: [9fans] IRQs again...

[andrey mirtchovski]

I forgot to mention that if we attach a R4 hard drive to the box
with LinuxBios on then we boot fine.  The IRQs are of course not
set correctly (i.e.  either all devices have IRQ==0 or some other
incoherent arrangement).

We want to have the kernel boot IL though, and that we cannot
accomplish because the kernel does not receive the interrupts from the
ethernet card.

If we follow the IRQ arrangement made by a real BIOS then we are able
to see _a_ DHCP request being sent out from the box, we see the DHCP
reply sent by the server, but the kernel never gets a note about it.

Re: [9fans] IRQs again...

[andrey mirtchovski]

[-- Attachment #1: Type: text/plain, Size: 535 bytes --]

Yes, plan9 boots fine (LinuxBios is on a separate disk on chip, that
we can substitute with the original BIOS anytime we want.  That's
where we get things like conf.npage0, conf.npage1, etc, which P9 does
not calculate correctly if there's no BIOS)...

As it currently stands, on two different types of southbridges (cyrix
and ALI) irregardless of how configured (or misconfigured) the IRQs
end up, we're unable to get an interrupt at all...

Next step is comparing 8259's registers with the ones from a normally
booted p9.

[-- Attachment #2: Type: message/rfc822, Size: 2310 bytes --]

From: jmk@plan9.bell-labs.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] IRQs again...
Date: Mon, 27 May 2002 00:23:18 -0400
Message-ID: <438ad8f565831fbb21a2cd36862b152c@plan9.bell-labs.com>

On Fri May 24 22:27:23 EDT 2002, rminnich@lanl.gov wrote:
> ...
> sooner or later, with hot plug happening, the OSes will have to play this
> fun game. LinuxBIOS got us there early in this case. But I think doing
> interrupt allocation and setup in the OS is inevitable.
> ...

We've been there and done that, both running the Plan 9 code base
on hot-plug systems and using a Plan 9 kernel as the boot ROM (is it
really almost 10 years ago?).

Before you flashed LinuxBIOS on the motherboard, is it likely Plan 9
would have booted?

Re: [9fans] IRQs again...

[jmk]

On Fri May 24 22:27:23 EDT 2002, rminnich@lanl.gov wrote:
> ...
> sooner or later, with hot plug happening, the OSes will have to play this
> fun game. LinuxBIOS got us there early in this case. But I think doing
> interrupt allocation and setup in the OS is inevitable.
> ...

We've been there and done that, both running the Plan 9 code base
on hot-plug systems and using a Plan 9 kernel as the boot ROM (is it
really almost 10 years ago?).

Before you flashed LinuxBIOS on the motherboard, is it likely Plan 9
would have booted?

Re: [9fans] IRQs again...

[Ronald G Minnich]

On Fri, 24 May 2002, David Gordon Hogan wrote:

> I've looked at the Linux code.  They actually use a heuristic to allocate
> IRQs, giving preference to ones that are unlikely to be used by something
> else.  Frankly, I'm scared to go there!  All of our machines have a BIOS,
> and it has not been necessary for us to do our own IRQ allocation.  But
> is this good enough?

sooner or later, with hot plug happening, the OSes will have to play this
fun game. LinuxBIOS got us there early in this case. But I think doing
interrupt allocation and setup in the OS is inevitable.

ron

Re: [9fans] IRQs again...

[David Gordon Hogan]

> Then the code tries to set the IRQs according to the pci routing
> table, and that's where the problems start.  The ali_get() routine
> seems to be very similar to the one linux uses, with several
> exceptions you can see below:
>
> 	plan9:ali_get():
> 		pirq = pcicfgr8(router, 0x48 + ((link-1)>>1));
> 		return (link & 1)? map[pirq&15]: map[pirq>>4];
>
> 	linux:pirq_ali_get():
>  	       return irqmap[read_config_nybble(router, 0x48, pirq-1)];
>
> I'm not sure what warrants these changes in plan9, but as an effect
> ali_get() returns '0' as the interrupt for all pci devices it is
> executed for.

The code fragments are equivalent.  read_config_nybble() is something
the Linux code defines to avoid duplicating all that nasty shifting and anding.

ali_get() is returning 0 on Plan 9 because that's what the southbridge
has in it.  As you are running without any BIOS, the IRQ routing in
the south bridge hasn't been set up.  We do not handle this situation
at present, we assume that the BIOS has allocated IRQs to the various
devices.  We only attempt to repair inconsistent IRQs, not allocate
them ourselves.

> The significant difference was that according to this specification
> the IRQs are to be obtained from the last two bytes of the info
> structure (pciinfo is the name linux and bsd use for it).  The first
> byte is passed as 'link' to ali_get() in the plan9 code (m[0] is the
> variable), however m[1] and m[2] form another 16 bit variable, having
> 1 in the place of each IRQ which is routable on the board (1eb8 for
> our test motherboard).  Going through the bits in search for ones
> gives us the interrupts we can use.

We should probably be checking those masks, at least as a sanity check.

> We're not particularly sure this is the proper way of doing things
> (both linux and freebsd seem to be doing it at one stage or another of
> the pci routing code, but...).

I've looked at the Linux code.  They actually use a heuristic to allocate
IRQs, giving preference to ones that are unlikely to be used by something
else.  Frankly, I'm scared to go there!  All of our machines have a BIOS,
and it has not been necessary for us to do our own IRQ allocation.  But
is this good enough?