[9fans] Interrupt saturation

[9fans] Interrupt saturation

[lucio]

The workstation I'm using presently seems to trigger as many
interrupts as stats(1) can display, while the syscall graph is also
extremely busy.

I presume this is anomalous.  Killing the single instance of
timesync(1) does not seem to make any difference at all.

Any suggestions to where I should look?  I'm not running anything that
would make me suspicious, but I'm going to restart the system just to
check.

++L

Re: [9fans] Interrupt saturation

[erik quanstrom]

On Sun Feb 28 04:57:09 EST 2010, lucio@proxima.alt.za wrote:
> The workstation I'm using presently seems to trigger as many
> interrupts as stats(1) can display, while the syscall graph is also
> extremely busy.
>
> I presume this is anomalous.  Killing the single instance of
> timesync(1) does not seem to make any difference at all.
>
> Any suggestions to where I should look?  I'm not running anything that
> would make me suspicious, but I'm going to restart the system just to
> check.

the exact number would be interesting.  i've added a field
to '#P/irqalloc' on my kernels that is the count of interrupts
per vector:

; cat '#P/irqalloc'
          3           0                    0 debugpt
          7           0                    0 mathemu
          8           0                    0 doublefault
          9           0                    0 mathover
         14           0                    0 fault386
         15           0                    0 unexpected
         16           0                    0 matherror
         50          18          18822033321 clock
         51          19                    0 lapicerror
         63          31                    0 lapicspurious
         65           1                    2 kbd
         73           9            772061679 ether0
         81          10              8689592 ether1
         89          11             11583013 ether2
         97           4                 5092 COM1
        105          14                    5 sdC (ata)

perhaps this would be useful.  you can pick it out
of ftp://ftp.quanstro.net/other/kernel.mkfs.bz2
in pc/trap.c.

- erik

Re: [9fans] Interrupt saturation

[ron minnich]

On Sun, Feb 28, 2010 at 1:55 AM,  <lucio@proxima.alt.za> wrote:
> The workstation I'm using presently seems to trigger as many
> interrupts as stats(1) can display, while the syscall graph is also
> extremely busy.

workstation only? I.e. it is not running venti?

ron

Re: [9fans] Interrupt saturation

[lucio]

> workstation only? I.e. it is not running venti?

Thanks for the chance to follow up.  I implemented quanstro's
enhancement to /proc, and this is what I get right now, some
twenty-four hours after starting up:

          3           0                    0 debugpt
          7           0                    0 mathemu
          8           0                    0 doublefault
          9           0                    0 mathover
         14           0                    0 fault386
         15           0                    0 unexpected
         16           0                    0 matherror
         50          18             76160483 clock
         51          19                    0 lapicerror
         63          31                    0 lapicspurious
         65           1                15761 kbd
         73          11              9799735 ether0
         81           6                    0 floppy
         89           3               124777 ac97audio
         97           5                    0 usbuhci
        105           9                    0 usbuhci
        113          11                    0 usbuhci
        121          10                    0 usbehci
        129          15                   82 sdD (ata)
        137           7                    0 lpt
        145          12               213171 kbdaux

The stuff Erik displayed looked a lot worse, so maybe it's just that
I'm not used to having stats(1) display a solid block for the
Interrupts.  Still, I wish cpu cycles could be better utilised.

And, tes, it's effectively a diskless terminal.

++L

Re: [9fans] Interrupt saturation

[erik quanstrom]

On Mon Mar  1 12:53:59 EST 2010, lucio@proxima.alt.za wrote:
> > workstation only? I.e. it is not running venti?
>
> Thanks for the chance to follow up.  I implemented quanstro's
> enhancement to /proc, and this is what I get right now, some
> twenty-four hours after starting up:
>
>          50          18             76160483 clock
>          65           1                15761 kbd
>          73          11              9799735 ether0
>          89           3               124777 ac97audio
>         145          12               213171 kbdaux
>
> The stuff Erik displayed looked a lot worse, so maybe it's just that
> I'm not used to having stats(1) display a solid block for the
> Interrupts.  Still, I wish cpu cycles could be better utilised.
>
> And, tes, it's effectively a diskless terminal.

my machine has been up for 90+ days.

looks like you just have HZ set to 1000.
 stats isn't counting on that.

also, 113 ethernet interrupts/sec is a pretty
good clip.

- erik

Re: [9fans] Interrupt saturation

[lucio]

> my machine has been up for 90+ days.
>
That sure makes a difference :-)

> looks like you just have HZ set to 1000.
>  stats isn't counting on that.
>
How does one change that?

> also, 113 ethernet interrupts/sec is a pretty
> good clip.

Yes, that's also a bit steep.  Intel chip, maybe I should install the
3Com card I had on the old hardware.  I changed from a 900MHz board to
a newer, 2.6GHz device with some (additional) peripherals built in.
Didn't expect a weird stats display.

++L

Re: [9fans] Interrupt saturation

[erik quanstrom]

> > looks like you just have HZ set to 1000.
> >  stats isn't counting on that.
> >
> How does one change that?

i wouldn't suspect that one would want to.
the higher clock rate allows more precise
scheduling.  the extra overhead should be
unnoticable on a 2.6ghz processor, except via
stats.

> > also, 113 ethernet interrupts/sec is a pretty
> > good clip.
>
> Yes, that's also a bit steep.  Intel chip, maybe I should install the
> 3Com card I had on the old hardware.  I changed from a 900MHz board to
> a newer, 2.6GHz device with some (additional) peripherals built in.
> Didn't expect a weird stats display.

i would imagine that you're getting this kind of
interrupt load from the nic because you're getting
a lot of packets.

do you suspect the interrupt load is a problem, or
are you just bothered by stats?

- erik

Re: [9fans] Interrupt saturation

[lucio]

> do you suspect the interrupt load is a problem, or
> are you just bothered by stats?

The latter, I replaced the workstation hardware bacause of a power
supply fan failure and the most noticeable difference was stats'
behaviour.

In passing, has anyone looked at fixing the uneven width of the last
column in a multi-host stats display?  I tried, but it seemed such a
convoluted computation I couldn't figure out how the last column's
width turned out a good few pixels wider than the others.  This is
particularly visible with many narrow columns (I have had up to eight
of them).

++L