[9fans] plan9: how to get irq to mach routing

[9fans] plan9: how to get irq to mach routing

[p9 newbie]

Hi, 

I would like to retrieve what  IRQs are routed to what Machs (or lapics).  I dumped IOAPIC's Redirection Table entries, but  Destination bits [63:48] are always '0xFF'.  Is there way to determine the routing between IRQs and Machs/LAPICs serving them?.  One rudimentary way I tried is  that, in the interrupt service routine I accessed the 'm' (Mach *) external variable to figure the routing between the IRQ and and the Mach..  Wondering if there is a better way. (for all legacy, msi and msi-x interrupt types)

Thanks
rg

Re: [9fans] plan9: how to get irq to mach routing

[erik quanstrom]

On Thu Jan 24 05:09:23 EST 2013, rgandhasri@gmail.com wrote:
> Hi,
> 
> I would like to retrieve what IRQs are routed to what Machs (or
> lapics).  I dumped IOAPIC's Redirection Table entries, but Destination
> bits [63:48] are always '0xFF'.  Is there way to determine the routing
> between IRQs and Machs/LAPICs serving them?.  One rudimentary way I
> tried is that, in the interrupt service routine I accessed the 'm'
> (Mach *) external variable to figure the routing between the IRQ and
> and the Mach..  Wondering if there is a better way.  (for all legacy,
> msi and msi-x interrupt types)

leaving 8259 interrupts to the side, in a system using apics, there are
four potential sources of interrupts for a processor
- processor traps,
- the lapic.  1 lapic attached to each processor
- i/o apics
- device sending an msi/msi-x interrupt.

obviously lapic interrupts (such as the clock timer) and traps
(such as #PF) are handled by the processor that generated them.
in flat or cluster mode, external interrupts are usually routed to
the "lowest priority" processor.  in physical mode, a particular
processor is targeted.  msi reuses the same logic.

the distribution uses physical addressing.  this means that interrupt
ι always targets processor p.  so the high bits should
match the processor picked.  see
	/n/sources/plan9/sys/src/9/pc/mp.c:793
	/n/sources/plan9/sys/src/9/pc/mp.c:824
i have not run this code to check, but it doesn't look like it sets
the redirection to broadcast.  (iirc, it's illegal to target a non-existent
processor from an i/o apic.)

i was having trouble with apics a few years ago due to the hardware
and thus did the apic implementation described in the man page.
this implementation supports flat, cluster & physical modes:

	http://www.quanstro.net/magic/man2html/3/apic

there are a number of files in '#P' that should help you poke
around such as '#P/mpirq' which prints out the vectors as
programmed.

nix uses the same implementation with msi, but only
uses physical mode in round-robin fashion.  i need to
merge these.  there's no reason for them to be different.

here's an example from a machine with physical-mode
vectors.  the i/o apic targets lapic 00 and 04.  
minooka;  cat '#P/mpvec' | grep -v '10000$'
           8           1 0000000000000141
           8           4 0400000000000161

here are the processors so we can find them.
(this format keeps changing.  don't depend on it.  :))

minooka; cat '#P/mpapic'
proc            0 00000000 00000000 00000000 be            0 0xfee00000
proc            1 01000000 01000000 01000000  e            8 0xfee00000
proc            2 02000000 02000000 02000000  e            1 0xfee00000
proc            3 03000000 03000000 03000000  e            9 0xfee00000
proc            4 04000000 04000000 04000000  e            2 0xfee00000
proc            5 05000000 05000000 05000000  e           10 0xfee00000
proc            6 06000000 06000000 06000000  e            3 0xfee00000
proc            7 07000000 07000000 07000000  e           11 0xfee00000
ioapic          8 00000000 00000000 00000000  e            0 0xfec00000
ioapic          9 00000000 00000000 00000000  e            0 0xfec8a000
proc           10 10000000 10000000 10000000  e            4 0xfee00000
proc           11 11000000 11000000 11000000  e           12 0xfee00000
proc           12 12000000 12000000 12000000  e            5 0xfee00000
proc           13 13000000 13000000 13000000  e           13 0xfee00000
proc           14 14000000 14000000 14000000  e            6 0xfee00000
proc           15 15000000 15000000 15000000  e           14 0xfee00000
proc           16 16000000 16000000 16000000  e            7 0xfee00000
proc           17 17000000 17000000 17000000  e           15 0xfee00000

so this is processor 0 and 4.  (that's not normally the case.  this
machine is pretty easy.)

for msi,

minooka;  /mnt/term/386/bin/pci -m | grep 00000000fee
#	flags	target addr	data		next ptr
0.31.2	0009	00000000fee05000	00004069	80
1.0.0	0181	00000000fee02000	00004051	50
1.0.1	0181	00000000fee03000	00004059	50
3.0.0	0081	00000000fee01000	00004049	44

bits 16:12 are the target.  so we see that we've targeted lapic 1, 3, 2, 5.
(which are fortunately for this example, identity mapped.)

- erik

Re: [9fans] plan9: how to get irq to mach routing

[p9 newbie]

Thanks for the detailed explanation Eriq. I couldn't find a pci binary that supported -m option.
I have 
# pci -help
usage: pci [-vb] [vid/did ...]

Thanks
rg

On Thursday, January 24, 2013 8:22:16 AM UTC-8, erik quanstrom wrote:
> On Thu Jan 24 05:09:23 EST 2013, rgandhasri@gmail.com wrote:
> 
> > Hi,
> 
> > 
> 
> > I would like to retrieve what IRQs are routed to what Machs (or
> 
> > lapics).  I dumped IOAPIC's Redirection Table entries, but Destination
> 
> > bits [63:48] are always '0xFF'.  Is there way to determine the routing
> 
> > between IRQs and Machs/LAPICs serving them?.  One rudimentary way I
> 
> > tried is that, in the interrupt service routine I accessed the 'm'
> 
> > (Mach *) external variable to figure the routing between the IRQ and
> 
> > and the Mach..  Wondering if there is a better way.  (for all legacy,
> 
> > msi and msi-x interrupt types)
> 
> 
> 
> leaving 8259 interrupts to the side, in a system using apics, there are
> 
> four potential sources of interrupts for a processor
> 
> - processor traps,
> 
> - the lapic.  1 lapic attached to each processor
> 
> - i/o apics
> 
> - device sending an msi/msi-x interrupt.
> 
> 
> 
> obviously lapic interrupts (such as the clock timer) and traps
> 
> (such as #PF) are handled by the processor that generated them.
> 
> in flat or cluster mode, external interrupts are usually routed to
> 
> the "lowest priority" processor.  in physical mode, a particular
> 
> processor is targeted.  msi reuses the same logic.
> 
> 
> 
> the distribution uses physical addressing.  this means that interrupt
> 
> ι always targets processor p.  so the high bits should
> 
> match the processor picked.  see
> 
> 	/n/sources/plan9/sys/src/9/pc/mp.c:793
> 
> 	/n/sources/plan9/sys/src/9/pc/mp.c:824
> 
> i have not run this code to check, but it doesn't look like it sets
> 
> the redirection to broadcast.  (iirc, it's illegal to target a non-existent
> 
> processor from an i/o apic.)
> 
> 
> 
> i was having trouble with apics a few years ago due to the hardware
> 
> and thus did the apic implementation described in the man page.
> 
> this implementation supports flat, cluster & physical modes:
> 
> 
> 
> 	http://www.quanstro.net/magic/man2html/3/apic
> 
> 
> 
> there are a number of files in '#P' that should help you poke
> 
> around such as '#P/mpirq' which prints out the vectors as
> 
> programmed.
> 
> 
> 
> nix uses the same implementation with msi, but only
> 
> uses physical mode in round-robin fashion.  i need to
> 
> merge these.  there's no reason for them to be different.
> 
> 
> 
> here's an example from a machine with physical-mode
> 
> vectors.  the i/o apic targets lapic 00 and 04.  
> 
> minooka;  cat '#P/mpvec' | grep -v '10000$'
> 
>            8           1 0000000000000141
> 
>            8           4 0400000000000161
> 
> 
> 
> here are the processors so we can find them.
> 
> (this format keeps changing.  don't depend on it.  :))
> 
> 
> 
> minooka; cat '#P/mpapic'
> 
> proc            0 00000000 00000000 00000000 be            0 0xfee00000
> 
> proc            1 01000000 01000000 01000000  e            8 0xfee00000
> 
> proc            2 02000000 02000000 02000000  e            1 0xfee00000
> 
> proc            3 03000000 03000000 03000000  e            9 0xfee00000
> 
> proc            4 04000000 04000000 04000000  e            2 0xfee00000
> 
> proc            5 05000000 05000000 05000000  e           10 0xfee00000
> 
> proc            6 06000000 06000000 06000000  e            3 0xfee00000
> 
> proc            7 07000000 07000000 07000000  e           11 0xfee00000
> 
> ioapic          8 00000000 00000000 00000000  e            0 0xfec00000
> 
> ioapic          9 00000000 00000000 00000000  e            0 0xfec8a000
> 
> proc           10 10000000 10000000 10000000  e            4 0xfee00000
> 
> proc           11 11000000 11000000 11000000  e           12 0xfee00000
> 
> proc           12 12000000 12000000 12000000  e            5 0xfee00000
> 
> proc           13 13000000 13000000 13000000  e           13 0xfee00000
> 
> proc           14 14000000 14000000 14000000  e            6 0xfee00000
> 
> proc           15 15000000 15000000 15000000  e           14 0xfee00000
> 
> proc           16 16000000 16000000 16000000  e            7 0xfee00000
> 
> proc           17 17000000 17000000 17000000  e           15 0xfee00000
> 
> 
> 
> so this is processor 0 and 4.  (that's not normally the case.  this
> 
> machine is pretty easy.)
> 
> 
> 
> for msi,
> 
> 
> 
> minooka;  /mnt/term/386/bin/pci -m | grep 00000000fee
> 
> #	flags	target addr	data		next ptr
> 
> 0.31.2	0009	00000000fee05000	00004069	80
> 
> 1.0.0	0181	00000000fee02000	00004051	50
> 
> 1.0.1	0181	00000000fee03000	00004059	50
> 
> 3.0.0	0081	00000000fee01000	00004049	44
> 
> 
> 
> bits 16:12 are the target.  so we see that we've targeted lapic 1, 3, 2, 5.
> 
> (which are fortunately for this example, identity mapped.)
> 
> 
> 
> - erik

Re: [9fans] plan9: how to get irq to mach routing

[erik quanstrom]

On Fri Jan 25 04:59:21 EST 2013, rgandhasri@gmail.com wrote:
> Thanks for the detailed explanation Eriq. I couldn't find a pci binary that supported -m option.
> I have
> # pci -help
> usage: pci [-vb] [vid/did ...]

there aren't any long options in plan 9, so the old way to say
that was "pci -?".  otherwise that should be parsed as 4 options,
all of which might be valid.

i put /n/sources/contrib/quanstor/8.pci up on sources.  the
man page is http://www.quanstro.net/magic/man2html.8/pci
it requires that the caller be allowed to read $#/pci/$dev^raw.
this seems safe enough.

- erik