[9fans] High interruptload on cpu server

[9fans] High interruptload on cpu server

[Matthias Teege]

Moin,

I have a cpu server running in a vmware machine. I run stats on my
terminal (also vmware) and on the cpu server and since some days
there is a relativ high interrupt/sec load on the cpu server. The
stats line remains on 50% of the display. All other stats are
"normal". The interrupt load on the terminal is much smaller.

Is there any way to detect, how is eating the interrupts? There
isn't much load on the cpu server.

Matthias

Re: [9fans] High interruptload on cpu server

[mirtchov]

you can use kprof(3) to profile the kernel, usually the interrupt
routine of the device driver for the device generating the interrupts
will be one with a higher percentage.

andrey

> Moin,
>
> I have a cpu server running in a vmware machine. I run stats on my
> terminal (also vmware) and on the cpu server and since some days
> there is a relativ high interrupt/sec load on the cpu server. The
> stats line remains on 50% of the display. All other stats are
> "normal". The interrupt load on the terminal is much smaller.
>
> Is there any way to detect, how is eating the interrupts? There
> isn't much load on the cpu server.
>
> Matthias

Re: [9fans] High interruptload on cpu server

[Matthias Teege]

mirtchov@cpsc.ucalgary.ca writes:

> you can use kprof(3) to profile the kernel, usually the interrupt
> routine of the device driver for the device generating the interrupts
> will be one with a higher percentage.

Nice. First "snapshoot" looks like this.

ms	  %	sym
65830	 96.5	halt
860	  1.2	i8259isr
280	  0.4	resrcwait
280	  0.4	wbflush
150	  0.2	genrandom
150	  0.2	addclock0link
120	  0.1	syscall
80	  0.1	validname
70	  0.1	putmmu
60	  0.0	newrgrp

Halt?

Matthias

Re: [9fans] High interruptload on cpu server

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 111 bytes --]

That means that your system is idle 96.5% of the time.
We actually execute the halt instruction to save power.

[-- Attachment #2: Type: message/rfc822, Size: 2832 bytes --]

From: Matthias Teege <matthias-9fans@mteege.de>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] High interruptload on cpu server
Date: Sun, 23 Nov 2003 18:29:02 +0100
Message-ID: <86u14v6ljf.fsf@gic.mteege.de>

mirtchov@cpsc.ucalgary.ca writes:

> you can use kprof(3) to profile the kernel, usually the interrupt
> routine of the device driver for the device generating the interrupts
> will be one with a higher percentage.

Nice. First "snapshoot" looks like this.

ms	  %	sym
65830	 96.5	halt
860	  1.2	i8259isr
280	  0.4	resrcwait
280	  0.4	wbflush
150	  0.2	genrandom
150	  0.2	addclock0link
120	  0.1	syscall
80	  0.1	validname
70	  0.1	putmmu
60	  0.0	newrgrp

Halt?

Matthias

Re: [9fans] High interruptload on cpu server

[boyd, rounin]

> That means that your system is idle 96.5% of the time.
> We actually execute the halt instruction to save power.

oh, the 'wait for interrupt' trick.  nice.

Re: [9fans] High interruptload on cpu server

[Fco.J.Ballesteros]

> Halt?
>
> Matthias

idle.

Re: [9fans] High interruptload on cpu server

[David Presotto]

If you're on a PC, you can look to see how much time is
spent in interrupts by 'acid -k $pid' and dumping the
intrtimes array.  Take a look at trap.c and you'll
understand.  It's just an array of 20 buckets for each
possible interrupt level.  Each bucket represents a
100 µ second interval.  You'll be able to tell if any
one level is hogging a lot of time.  I've debated adding
a user readable interface but then I'ld have to write a
man page.

kprof is moderately crappy for timing interrupts since
it samples in the clock routine which is splhi'd as
are the other interrupt routines, i.e., sampling is
mutually exclusive to the interrupt routine running.

Re: [9fans] High interruptload on cpu server

[Matthias Teege]

David Presotto <presotto@closedmind.org> writes:

> If you're on a PC, you can look to see how much time is
> spent in interrupts by 'acid -k $pid' and dumping the
> intrtimes array.  Take a look at trap.c and you'll
> understand.  It's just an array of 20 buckets for each

Ok, I'll do so. I discover some nice thing. I can bring down the
interrupt load to a normal level (99% idle ;-)) if I run a nmap
against the cpu server. After the nmap run the interrupt load stays
on that normal level. Maybe I kill a hanging service (timesync?)
whit nmap.

Funny.
Matthias

--
Matthias Teege -- http://www.mteege.de
make world not war

Re: [9fans] High interruptload on cpu server

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

Depends on what you mean by kill.  Just removing the file from
/rc/bin/service shouldn't be a problem.  Listen will fork and
exec it, and the exec will just die.

Could you just probe one service at a time and see what's fixing
your interrupt load?

[-- Attachment #2: Type: message/rfc822, Size: 2957 bytes --]

From: Matthias Teege <matthias-9fans@mteege.de>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] High interruptload on cpu server
Date: Tue, 25 Nov 2003 14:32:02 +0100
Message-ID: <864qws376f.fsf@gic.mteege.de>

David Presotto <presotto@closedmind.org> writes:

> If you're on a PC, you can look to see how much time is
> spent in interrupts by 'acid -k $pid' and dumping the
> intrtimes array.  Take a look at trap.c and you'll
> understand.  It's just an array of 20 buckets for each

Ok, I'll do so. I discover some nice thing. I can bring down the
interrupt load to a normal level (99% idle ;-)) if I run a nmap
against the cpu server. After the nmap run the interrupt load stays
on that normal level. Maybe I kill a hanging service (timesync?)
whit nmap.

Funny.
Matthias

--
Matthias Teege -- http://www.mteege.de
make world not war

Re: [9fans] High interruptload on cpu server

[Matthias Teege]

David Presotto <presotto@closedmind.org> writes:

> Could you just probe one service at a time and see what's fixing
> your interrupt load?

Yes, it is port 22 ssh. If I connect to 22 via telnet,ssh, or nmap
the interrupt load goes down for a while. There must be somthing
wrong in my configuration because I can't connect via ssh. I only
get

debug1: ssh_connect: needpriv 0
debug1: Connecting to coco.mteege.de [192.168.153.21] port 22.
debug1: Connection established.
ssh_exchange_identification: Connection closed by remote host
debug1: Calling cleanup 0x804c158(0x0)

Bis dann
Matthias

Re: [9fans] High interruptload on cpu server

[mirtchov]

> David Presotto <presotto@closedmind.org> writes:
>
>> Could you just probe one service at a time and see what's fixing
>> your interrupt load?
>
> Yes, it is port 22 ssh. If I connect to 22 via telnet,ssh, or nmap
> the interrupt load goes down for a while. There must be somthing
> wrong in my configuration because I can't connect via ssh. I only
> get
>

did you run auth/rsagen?

Re: [9fans] High interruptload on cpu server

[Matthias Teege]

mirtchov@cpsc.ucalgary.ca writes:

> did you run auth/rsagen?

After reading the manpage --- yes. But I ask myself, Is there any
place to run auth/rsagen at boottime and is Plan9 ssh only protocol
version 1?

Matthias

Re: [9fans] High interruptload on cpu server

[William Josephson]

On Wed, Nov 26, 2003 at 07:56:32AM +0100, Matthias Teege wrote:
> After reading the manpage --- yes. But I ask myself, Is there any
> place to run auth/rsagen at boottime and is Plan9 ssh only protocol
> version 1?

Yes, SSH is version one only.  I've been meaning to push out
a version two implementation for about eighteen months but
have been too busy to clean it up.  If anyone is serious about
making it go, I can probably dust it off and give them the
majority of what I have...

Re: [9fans] High interruptload on cpu server

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

running auth/rsagen at boot time would be silly.  The ssh clients will expect your
host key not to change.  If it changes every time you boot, you'll just force people
to keep accepting new keys with no way to verify them.  If so, why bother since
anyone can masquerade as you.

[-- Attachment #2: Type: message/rfc822, Size: 2545 bytes --]

From: Matthias Teege <matthias-9fans@mteege.de>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] High interruptload on cpu server
Date: Wed, 26 Nov 2003 07:56:32 +0100
Message-ID: <86brqzsjm1.fsf@gic.mteege.de>

mirtchov@cpsc.ucalgary.ca writes:

> did you run auth/rsagen?

After reading the manpage --- yes. But I ask myself, Is there any
place to run auth/rsagen at boottime and is Plan9 ssh only protocol
version 1?

Matthias

Re: [9fans] High interruptload on cpu server

[mirtchov]

> After reading the manpage --- yes. But I ask myself, Is there any
> place to run auth/rsagen at boottime and is Plan9 ssh only protocol
> version 1?

put the key rsagen creates in secstore, if you look back in the
archives you'll see me asking the same question and getting the same
answer.  after creating a secstore on the auth server and adding the
rsagen contents in it there have been no issues with ssh-ing to the
machines.

andrey

ps: nobody uses ssh on our machines -- they all connect via drawterms
or import stuff like /mail directly from their own Plan 9
installation.  my latest rage is to boot a vmware session as a
terminal+cfs from home, connected to the university Plan 9 machines.
my network connection is capped at 150KB download and 60KB upload,
with latency of about 15ms to the university (same company provides
both mine and the university's internet connections), which is just
barely enough for a decent drawterm session, provided nothing else is
hogging the connection.

i've found term+cfs to be a perfect combination.  it stays up much
longer and allows me to edit text much better.  it sucks for compiling
stuff, but i can always cpu to the main server and do it there, even
in the same acme window

Re: [9fans] High interruptload on cpu server

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

It's quite odd.  The listener doesn't do anything while waiting for a call,
i.e., sshserve doesn't get started until the call is made.  No matter how
badly you misconfigured it, it won't do anything until you connect to
the port.  It sounds more like your interrupts stop when you have something
to run.

What hardware are you doing this on?  What does /dev/sysstat show
about the interrupt load?

[-- Attachment #2: Type: message/rfc822, Size: 2937 bytes --]

From: Matthias Teege <matthias-9fans@mteege.de>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] High interruptload on cpu server
Date: Tue, 25 Nov 2003 16:35:19 +0100
Message-ID: <86znek1mwi.fsf@gic.mteege.de>

David Presotto <presotto@closedmind.org> writes:

> Could you just probe one service at a time and see what's fixing
> your interrupt load?

Yes, it is port 22 ssh. If I connect to 22 via telnet,ssh, or nmap
the interrupt load goes down for a while. There must be somthing
wrong in my configuration because I can't connect via ssh. I only
get

debug1: ssh_connect: needpriv 0
debug1: Connecting to coco.mteege.de [192.168.153.21] port 22.
debug1: Connection established.
ssh_exchange_identification: Connection closed by remote host
debug1: Calling cleanup 0x804c158(0x0)

Bis dann
Matthias

Re: [9fans] High interruptload on cpu server

[Matthias Teege]

David Presotto <presotto@closedmind.org> writes:

> It's quite odd.  The listener doesn't do anything while waiting for a call,
> i.e., sshserve doesn't get started until the call is made.  No matter how
> badly you misconfigured it, it won't do anything until you connect to
> the port.  It sounds more like your interrupts stop when you have something
> to run.

CPU/auth Server and Term running under VMware. My new cpu server is
appointed.

/dev/sysstat look like that

Terminal:

0     2981455     4928785     3083886       44435           0           0         231          91           1

CPU:

0     2434644    17515256      969406       87557           0           0         303          93           4

Matthias

Re: [9fans] High interruptload on cpu server

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 451 bytes --]

The numbers are meaningless unless you look at them twice over some period and see what how
the numbers change.

That said, there doesn't seem to be much happening on either system.   If
you add up the last two numbers and subtract from 100, that's the ammount
of time spent elsewhere (probably vmware somewhere).

It sounds like you're being bitten by an interaction with vmware.  What
does a ps on the cpu server show?  Is it doing anything?

[-- Attachment #2: Type: message/rfc822, Size: 3073 bytes --]

From: Matthias Teege <matthias-9fans@mteege.de>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] High interruptload on cpu server
Date: Wed, 26 Nov 2003 07:54:25 +0100
Message-ID: <86fzgbsjpk.fsf@gic.mteege.de>

David Presotto <presotto@closedmind.org> writes:

> It's quite odd.  The listener doesn't do anything while waiting for a call,
> i.e., sshserve doesn't get started until the call is made.  No matter how
> badly you misconfigured it, it won't do anything until you connect to
> the port.  It sounds more like your interrupts stop when you have something
> to run.

CPU/auth Server and Term running under VMware. My new cpu server is
appointed.

/dev/sysstat look like that

Terminal:

0     2981455     4928785     3083886       44435           0           0         231          91           1

CPU:

0     2434644    17515256      969406       87557           0           0         303          93           4

Matthias

Re: [9fans] High interruptload on cpu server

[Rob Pike]

> Depends on what you mean by kill.  Just removing the file from
> /rc/bin/service shouldn't be a problem.  Listen will fork and
> exec it, and the exec will just die.

i'd just chmod 0 the file.

-rob