[9fans] killing rogue processes

[9fans] killing rogue processes

[mirtchov]

I have a few exportfs processes that I can't kill:


	octarine% echo kill > /proc/7207/ctl
	octarine% cat /proc/7207/status
	exportfs                    andrey                      Stat                  0	\
		     1085720     1081850           0          10           0	\
		      220          10           2

any suggestions?

Re: [9fans] killing rogue processes

[Russ Cox]

> I have a few exportfs processes that I can't kill:
>
>
> 	octarine% echo kill > /proc/7207/ctl
> 	octarine% cat /proc/7207/status
> 	exportfs                    andrey                      Stat                  0	\
> 		     1085720     1081850           0          10           0	\
> 		      220          10           2
>
> any suggestions?

Kill the server it is talking to, which is clearly misbehaving.

> libiconv...

Such a program already exists -- tcs.

Russ

Re: [9fans] killing rogue processes

[andrey mirtchovski]

On Thu, 18 Dec 2003, Russ Cox wrote:

> Kill the server it is talking to, which is clearly misbehaving.

alright, what's the easiest way to figure out which process is using port
XYZ?

Re: [9fans] killing rogue processes

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 118 bytes --]

netstat -n to get the tcp channel and grep tcp/channel /proc/*/fd.  I'm sure
you can do the shell script around that.

[-- Attachment #2: Type: message/rfc822, Size: 2459 bytes --]

From: andrey mirtchovski <mirtchov@cpsc.ucalgary.ca>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] killing rogue processes
Date: Thu, 18 Dec 2003 11:28:00 -0700 (MST)
Message-ID: <Pine.LNX.4.44.0312181127360.8078-100000@fbsd.cpsc.ucalgary.ca>

On Thu, 18 Dec 2003, Russ Cox wrote:

> Kill the server it is talking to, which is clearly misbehaving.

alright, what's the easiest way to figure out which process is using port
XYZ?

Re: [9fans] killing rogue processes

[mirtchov]

> netstat -n to get the tcp channel and grep tcp/channel /proc/*/fd.  I'm sure
> you can do the shell script around that.

I have an 'lsof' that lists all file descriptors + process #s.. just grepping for 'tcp'
gave me the clues...

I've even sent the code+man page to 9trouble, but don't know what happened :)

Re: [9fans] killing rogue processes

[Russ Cox]

> > netstat -n to get the tcp channel and grep tcp/channel /proc/*/fd.  I'm sure
> > you can do the shell script around that.
>
> I have an 'lsof' that lists all file descriptors + process #s.. just grepping for 'tcp'
> gave me the clues...
>
> I've even sent the code+man page to 9trouble, but don't know what happened :)

i don't remember the one you sent to 9trouble,
but it seems trivial enough not to warrant a
separate command:

#!/bin/rc
cd /proc
for (i in *)
	cat $i/fd | sed s/^/$i' '/

Re: [9fans] killing rogue processes

[mirtchov]

> i don't remember the one you sent to 9trouble,
> but it seems trivial enough not to warrant a
> separate command:
>
> #!/bin/rc
> cd /proc
> for (i in *)
> 	cat $i/fd | sed s/^/$i' '/

it's written in C for the same reason 'ps' is, whatever that may be :)

Re: [9fans] killing rogue processes

[Russ Cox]

> > i don't remember the one you sent to 9trouble,
> > but it seems trivial enough not to warrant a
> > separate command:
> >
> > #!/bin/rc
> > cd /proc
> > for (i in *)
> > 	cat $i/fd | sed s/^/$i' '/
>
> it's written in C for the same reason 'ps' is, whatever that may be :)

ps does non-trivial reformatting of the /proc files.

Re: [9fans] killing rogue processes

[ron minnich]

On Thu, 18 Dec 2003, Russ Cox wrote:

> ps does non-trivial reformatting of the /proc files.


yes but look what's possible with lsof:

lsof [ -?abChlnNOPRstUvVX ] [ -A A ] [ -c c ] [ +|-d d ] [ +|-D D ] [ +|-f
[cfgGn] ] [ -F [f] ] [ -g [s] ] [ -i [i] ] [ -k k ] [ +|-L [l] ] [ -m m ]
[ +|-M ] [ -o [o] ] [ -p s ] [ +| -r [t] ] [ -S [t] ] [ -T [t] ] [ -u s ]
[ +|-w ] [ -- ] [names]

yessirree, I'm sure we want an lsof with all them purty switches.

ron

Re: [9fans] killing rogue processes

[boyd, rounin]

> lsof [ -?abChlnNOPRstUvVX ] [ -A A ] [ -c c ] [ +|-d d ] [ +|-D D ] [ +|-f
> [cfgGn] ] [ -F [f] ] [ -g [s] ] [ -i [i] ] [ -k k ] [ +|-L [l] ] [ -m m ]
> [ +|-M ] [ -o [o] ] [ -p s ] [ +| -r [t] ] [ -S [t] ] [ -T [t] ] [ -u s ]
> [ +|-w ] [ -- ] [names]
>
> yessirree, I'm sure we want an lsof with all them purty switches.

yeah, don't i know it.  a decent /proc would have removed the need
for it (in the '90s) and /proc existed back in '83 or '84 iirc.

Re: [9fans] killing rogue processes

[mirtchov]

> #!/bin/rc
>> > cd /proc
>> > for (i in *)
>> > 	cat $i/fd | sed s/^/$i' '/

This one actually works:

	% cat lsof
	#!/bin/rc

	cd /proc
	for (i in *)
		if(! ~ $i trace)
			cat $i/fd | sed 's/^/'^$i^' /'
	%

Subst that for lsof.c if you wish -- simple is better :)

By the way, it wasn't a script because I was doing 'ls /proc' instead
of 'in *' and had to deal with the 'ls' process exiting before I had
gotten around to catting its fd file, not that I didn't try :)

andrey

Re: [9fans] killing rogue processes

[boyd, rounin]

> I have an 'lsof' that lists all file descriptors + process #s.. just
grepping for 'tcp'
> gave me the clues...

you ported that to plan 9?

nasty little bugger, that thing;  /dev/kmem city!

Re: [9fans] killing rogue processes

[mirtchov]

>> I have an 'lsof' that lists all file descriptors + process #s.. just
> grepping for 'tcp'
>> gave me the clues...
>
> you ported that to plan 9?

no, I wrote my own:

	octarine% wc -l $home/kern/cmd/lsof.c
	     89 /usr/andrey/kern/cmd/lsof.c
	octarine%

there's also top.  for now it is a perl script, but I have been
thinking of ways to make it look like stats a bit more (how would you
represent different chunks of processor time?  pie chart?), it's
another 45 lines of code and actually that's what helped me find what
was causing all the load on the system.

Re: [9fans] killing rogue processes

[Scott Schwartz]

| there's also top.  for now it is a perl script, but I have been
| thinking of ways to make it look like stats a bit more (how would you
| represent different chunks of processor time?  pie chart?),

For inspiration, check out lavaps: processes represented as bubbles in
a lava lamp.

http://www.isi.edu/~johnh/SOFTWARE/LAVAPS/

Re: [9fans] killing rogue processes

[mirtchov]

>
> http://www.isi.edu/~johnh/SOFTWARE/LAVAPS/

oh, no, if we're going to go overboard let's have psdoom:

	http://psdoom.sourceforge.net
	http://psdoom.sourceforge.net/images/ss2.gif

the bigger the monster the more cputime it has consumed :)

Re: [9fans] killing rogue processes

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 28 bytes --]

I updated lsof.c to sources.

[-- Attachment #2: Type: message/rfc822, Size: 2306 bytes --]

From: mirtchov@cpsc.ucalgary.ca
To: 9fans@cse.psu.edu
Subject: Re: [9fans] killing rogue processes
Date: Thu, 18 Dec 2003 13:29:34 -0700
Message-ID: <a8176ec3da9468ed7d247001954d2e44@plan9.ucalgary.ca>

>> I have an 'lsof' that lists all file descriptors + process #s.. just
> grepping for 'tcp'
>> gave me the clues...
>
> you ported that to plan 9?

no, I wrote my own:

	octarine% wc -l $home/kern/cmd/lsof.c
	     89 /usr/andrey/kern/cmd/lsof.c
	octarine%

there's also top.  for now it is a perl script, but I have been
thinking of ways to make it look like stats a bit more (how would you
represent different chunks of processor time?  pie chart?), it's
another 45 lines of code and actually that's what helped me find what
was causing all the load on the system.

Re: [9fans] killing rogue processes

[boyd, rounin]

> no, I wrote my own:

good call. should be pretty easy with plan 9 /proc (unlike lunix's).

i ported it [lsof] to some NCR Sys VileR4 (sic) boxes -- nasty.

Re: [9fans] killing rogue processes

[boyd, rounin]

> netstat -n to get the tcp channel and grep tcp/channel /proc/*/fd.  I'm
sure
> you can do the shell script around that.

or reach for your M9 ...

Re: [9fans] killing rogue processes

[Bruce Ellis]

----- Original Message -----
From: "Russ Cox" <rsc@swtch.com>
To: <9fans@cse.psu.edu>
Sent: Friday, December 19, 2003 3:57 AM
Subject: Re: [9fans] killing rogue processes


...

> Kill the server it is talking to, which is clearly misbehaving

no, you have to try harder...  sometimes a lot harder.
it is not a problem with misbehaving servers.  it's a flaw
in 9p(2000), or devmnt, take your pick.

when a process exits it closes its fds.
if the close is the final reference to a channel then
the chan is clunked and if the chan is being served
the poor old process sends off a Tclunk and sits
around waiting for the Rclunk.  if the server is also
exiting (which happens when they are in the same
killed pgrp) they might be lucky enough to *not* be
stuck waiting for each other.

a very common scenario is a deadly embrace between
rio upas/fs and exportfs.  it happens all the time to me.
rob's suggestion was to "rattle the cage enough" to get
things to go away.

things were even worse in inferno where fds are never
closed explicitly.  i fixed it in inferno because any
system of any complexity demanded a huge stick to
rattle the cage.

as for "who is using inferno", well i use it for almost
everything - almost all code i've written this year has
been in limbo.  i run emu on plan9 for development,
emu on win2k to give me access to all the devices
that i can't get data sheets for (particularly usb midi
and non-standard usb audio).  of course it also runs
in my phone and router (wrt54g) and on $50 386
machines from china, various arm/xscale boards etc.
my inferno web server (with active scripting) is so
much smaller and faster than apache that it runs
without pain on all of the above platforms.

i've had informal sporadic discussions with VN about
how to share the wealth (the inferno i run is so divergent
from theirs that i wouldn't attempt a merge).  maybe
something will happen - when the weather gets worse.

just my threepence worth.

brucee