[9fans] irq latency

[9fans] irq latency

[erik quanstrom]

tl;dr: pcie access = latency

it's interesting to take a look at irq latency for a number of devices on different
machines.  it looks like the experimental device --- the interrupt does nothing ---
gives us a lower bound for irq latency, which works out to be 36ns (!).  clearly
this doesn't count dispatch, or the overhead of trap().

m# aux/cpuid -i
Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz
vector.mach     bus irq	count             sum(cycles)  cycles/call  type  name
m# /usr/quanstro/bin/rc/irqlatency
       66.0          11                24605              3143914 127.78    msi-x experimental
       50.0          50               348487            802964226 2304.14    lapic APIC timer
       50.6          50               324950            841831354 2590.65    lapic APIC timer
       50.4          50               319746            832771958 2604.48    lapic APIC timer
       50.2          50               323061            846873676 2621.40    lapic APIC timer
       50.3          50               323080            867020068 2683.61    lapic APIC timer
       50.5          50               323056            886385231 2743.75    lapic APIC timer
       50.7          50               324165            905699976 2793.95    lapic APIC timer
       50.1          50               323519            997341922 3082.79    lapic APIC timer
       65.0          11                 4383             45452712 10370.23    msi-x ether0
       66.5           5                66222            777537556 11741.38   ioapic usbehci
       65.7          11                    9               247991 27554.56      msi sdF (ahci)
       65.1           4                 1084             38400923 35425.21   ioapic COM1

; aux/cpuid -i
AMD Phenom(tm) II X4 965 Processor
; irqlatency
       50.3          50             83106879         109878207918 1322.13    lapic APIC timer
       50.2          50             83070335         109844976915 1322.31    lapic APIC timer
       50.1          50             83228230         110448926024 1327.06    lapic APIC timer
       50.0          50             90772899         169367434384 1865.84    lapic APIC timer
       65.2          10              1179147          11639594649 9871.20    msi-x ether0
       65.0           1                17657            178037258 10083.10   ioapic i8042
       68.2          12               312539           3476219928 11122.52   ioapic kbdaux
       66.0           4                  911             21026234 23080.39   ioapic COM1
       71.1          11                   58              1780132 30691.93      msi sdE (ahci)
       68.1          10                    1              7934988 7934988.00   ioapic usbohci

given the efficency of locks, and the same ethernet devices (i211), we can reason
that since the latency is prop. to the frequency,

	10370*3.4/3.5 = 10073.71428571

the latency must largely be pcie register reads and writes.  sadly, the msi-x interrupt
doesn't handle single causes, otherwise no register access would be necessary.

- erik

ps.

interesting question: why do the lapic timers take longer on the faster machine?  that's
double the latency.  is this an intel thing?  (note this is a different ethernet chipset,
and a different driver so the comparison with the other two is bogus.)

lilly; aux/cpuid -i
         Intel(R) Atom(TM) CPU D525   @ 1.80GHz
lilly; irqlatency
       50.3          50            802180313        1705456340928 2126.03    lapic APIC timer
       50.1          50            803988180        1811997635679 2253.76    lapic APIC timer
       50.2          50            832505288        1914666440067 2299.89    lapic APIC timer
       50.0          50            830861969        2509217805168 3020.02    lapic APIC timer
       70.2          15                    2                16587 8293.50   ioapic usbehci
       66.2          11             36660721         362749665600 9894.78      msi ether0
       66.1          10               395700           4099132008 10359.19      msi ether1
       65.0           1                    1                10404 10404.00   ioapic i8042
       68.1          14                   35               379197 10834.20   ioapic usbuhci
       67.3          10                   95              1065114 11211.73   ioapic usbehci
       65.3           4                  603              8503128 14101.37   ioapic COM1

Re: [9fans] irq latency

[erik quanstrom]

On Mon Sep  7 18:24:05 PDT 2015, quanstro@quanstro.net wrote:
> tl;dr: pcie access = latency

speaking of latency, 9fans seems to have >6hr latency right now.

- erik