[9fans] multiple pings cause panic

[9fans] multiple pings cause panic

[James Chapman]

On my new plan 9 terminal running under vmware fusion on my leopard
macbook this happens:

Executing the command

term% for (i in `{seq 1 254}) {ip/ping -n 1 192.168.1.1&}

causes

panic: Fsprotoclone: all conversations in use
panic :Fsprotoclone: all conversations in use
dumpstack disabled
cpu0: exiting

Regardless of whether my program was sensible I guess this isn't
supposed to happen. Right?

James

Re: [9fans] multiple pings cause panic

[erik quanstrom]

> term% for (i in `{seq 1 254}) {ip/ping -n 1 192.168.1.1&}

i think you mean

	for(i in `{seq 1 254}){ip/ping -n1 192.168.1.$i}

> panic: Fsprotoclone: all conversations in use
> panic :Fsprotoclone: all conversations in use
> dumpstack disabled
> cpu0: exiting
>
> Regardless of whether my program was sensible I guess this isn't
> supposed to happen. Right?

nope.  that's the way it's supposed to go.
the idea is that if all the conversations are used up,
then your machine is useless and you might as well
panic.

however, a fair question to ask is, is this limit
reasonable, and is panicing really the right thing
to do.

i'd guess printing nasties on the console makes
more sense.

- erik

Re: [9fans] multiple pings cause panic

[geoff]

There are limits on the number of concurrent conversations per
protocol.  For UDP, it's currently 1024, for TCP it's currently 1024
on terminals and 4096 on cpu servers.  For ICMP, it's currently 128;
you may want to raise that.  To find the current limits,

	grep '>nc = ' /sys/src/9/ip/*.c

Re: [9fans] multiple pings cause panic

[James Chapman]

On Sun, Apr 18, 2010 at 11:32 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> term% for (i in `{seq 1 254}) {ip/ping -n 1 192.168.1.1&}
>
> i think you mean
>
>        for(i in `{seq 1 254}){ip/ping -n1 192.168.1.$i}

Sorry, I meant

for(i in `{seq 1 254}){ip/ping -n1 192.168.1.$i&}

I added the & as I didn't want to wait for one to timeout/succeed
before trying the next one.

Is there a more sensible way to do this from rc? I.e. to kick off a
lot of pings at once and collect the results later.

>> panic: Fsprotoclone: all conversations in use
>> panic :Fsprotoclone: all conversations in use
>> dumpstack disabled
>> cpu0: exiting

> nope.  that's the way it's supposed to go.
> the idea is that if all the conversations are used up,
> then your machine is useless and you might as well
> panic.
>
> however, a fair question to ask is, is this limit
> reasonable, and is panicing really the right thing
> to do.

I was expecting that a panic was a bug and I shouldn't be able to cause one.

I'm running a standalone system here but if I wasn't could I bring
down the server like this or only my terminal?

Re: [9fans] multiple pings cause panic

[erik quanstrom]

> On Sun, Apr 18, 2010 at 11:32 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> >> term% for (i in `{seq 1 254}) {ip/ping -n 1 192.168.1.1&}
> >
> > i think you mean
> >
> >        for(i in `{seq 1 254}){ip/ping -n1 192.168.1.$i}
>
> Sorry, I meant
>
> for(i in `{seq 1 254}){ip/ping -n1 192.168.1.$i&}
>
> I added the & as I didn't want to wait for one to timeout/succeed
> before trying the next one.

i removed the & on purpose.  if you don't want to write
your own version of ping, doing them one-at-a-time
makes the most sense.

if you want to get super fancy, you could
	batch=10
	j=()
	for(i in `{seq 1 254}){
		if(! ~ $#j $batch){
			j=($i $j)
			ping -i1 192.168.0.$i&
		}
		if not{
			j=()
			wait
		}
	}
	if(! ~ $#j 0)
		wait

what are you *really* trying to do?  if you want to find servers
on the local segment, sending udp packets to them will fill arp
tables.  no timeout required.

> I was expecting that a panic was a bug and I shouldn't be able to cause one.
>
> I'm running a standalone system here but if I wasn't could I bring
> down the server like this or only my terminal?

sure.  that's why i said i think it can be debated if this is the
right response.  but it is definately intentional.  the problem is
that for some servers, running out of (say) tcp conversations
will make the server useless and a panic/reboot will set things
right, if we assume that the condition was temporary.

- erik

Re: [9fans] multiple pings cause panic

[erik quanstrom]

On Sun Apr 18 16:41:17 EDT 2010, geoff@plan9.bell-labs.com wrote:
> There are limits on the number of concurrent conversations per
> protocol.  For UDP, it's currently 1024, for TCP it's currently 1024
> on terminals and 4096 on cpu servers.  For ICMP, it's currently 128;
> you may want to raise that.  To find the current limits,
>
> 	grep '>nc = ' /sys/src/9/ip/*.c

it's surprisingly easy to burn through connections since connections
can hang out in Finwait2 for a long time, burning up packets and
connections:

	; cpu -c 'cd /net.alt/tcp; x=1??? echo $x($#x)'
	1175

i haven't had time to take a look at this issue, but i wonder if acks
during Finwait2 are resetting the timer.  this is definately related
to smtp &/| pop.

i sometimes use this script as hostowner to clear out these
connections

	x=`{grep -l Finwait2 */status|sed 's:/status::g'}
	for(i in $x)
		if(~ `{cat $i^/local} *!110 *!25)
			>$i^/ctl echo hangup

- erik

Re: [9fans] multiple pings cause panic

[cinap_lenrek]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

i dont think you need that. devip calls a protocol specific
garbage collector procedure that will close these hanging
connections for you if it runs out of connections.

/sys/src/9/ip/tcp.c:/^tcpgc

well, the test is a little bit more complicated as it takes
a minimum inactivity time into account.

made a patch that changes the panic into a print:

/n/sources/patch/devip-allconvused-panic

--
cinap

[-- Attachment #2: Type: message/rfc822, Size: 3087 bytes --]

From: erik quanstrom <quanstro@quanstro.net>
To: 9fans@9fans.net
Subject: Re: [9fans] multiple pings cause panic
Date: Sun, 18 Apr 2010 19:52:04 -0400
Message-ID: <ae24cc03d1ed549c7f22fea9af04a950@kw.quanstro.net>

On Sun Apr 18 16:41:17 EDT 2010, geoff@plan9.bell-labs.com wrote:
> There are limits on the number of concurrent conversations per
> protocol.  For UDP, it's currently 1024, for TCP it's currently 1024
> on terminals and 4096 on cpu servers.  For ICMP, it's currently 128;
> you may want to raise that.  To find the current limits,
>
> 	grep '>nc = ' /sys/src/9/ip/*.c

it's surprisingly easy to burn through connections since connections
can hang out in Finwait2 for a long time, burning up packets and
connections:

	; cpu -c 'cd /net.alt/tcp; x=1??? echo $x($#x)'
	1175

i haven't had time to take a look at this issue, but i wonder if acks
during Finwait2 are resetting the timer.  this is definately related
to smtp &/| pop.

i sometimes use this script as hostowner to clear out these
connections

	x=`{grep -l Finwait2 */status|sed 's:/status::g'}
	for(i in $x)
		if(~ `{cat $i^/local} *!110 *!25)
			>$i^/ctl echo hangup

- erik

Re: [9fans] multiple pings cause panic

[erik quanstrom]

> made a patch that changes the panic into a print:
>
> /n/sources/patch/devip-allconvused-panic

i'm not sure that's what one wants, in general.
depending on the protocol and the use case, you may
want to
(a) panic
(b) print (carefully)
(c) wait for the resource to become available.

i'd be afraid of transforming a simple, automaticly
recoverable problem into something that just wedges
the machine with prints to the console and no progress.

i've never seen a machine panic for this reason.

- erik

Re: [9fans] multiple pings cause panic

[cinap_lenrek]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

if one wants to retry connecting he can.  not being able to make a
network connection is a very common case so i guess it would be ok to
simply let it fail on this point instead of hanging and waiting.

rebooting sounds like a little bit too drastic for me. expecially if it
is about icmp sockets. devip just has too less context to make a good
decision.

the comment above the panic call also hinted that it was ment as some
kind of a bet or a debugging leftover so it felt save to remove it :)

anyway. the patch may or may not get applied or someone comes up
with a better solution. creating it was just a matter of seconds and i
apologize for blocking the patch tu^W^Wqueue if it caused any
inconvinience. :)

--
panic("me is a boofhead");

[-- Attachment #2: Type: message/rfc822, Size: 2618 bytes --]

From: erik quanstrom <quanstro@quanstro.net>
To: 9fans@9fans.net
Subject: Re: [9fans] multiple pings cause panic
Date: Sun, 18 Apr 2010 20:27:32 -0400
Message-ID: <ba655f19e2c16143d49aa4aa34cc1b9f@kw.quanstro.net>

> made a patch that changes the panic into a print:
>
> /n/sources/patch/devip-allconvused-panic

i'm not sure that's what one wants, in general.
depending on the protocol and the use case, you may
want to
(a) panic
(b) print (carefully)
(c) wait for the resource to become available.

i'd be afraid of transforming a simple, automaticly
recoverable problem into something that just wedges
the machine with prints to the console and no progress.

i've never seen a machine panic for this reason.

- erik

Re: [9fans] multiple pings cause panic

[andrey mirtchovski]

> panic("me is a boofhead");

no, that's presotto :p

Re: [9fans] multiple pings cause panic

[erik quanstrom]

On Sun Apr 18 21:21:10 EDT 2010, cinap_lenrek@gmx.de wrote:

> if one wants to retry connecting he can.

if you're primarly functioning as a server, that might be true,
but not helpful.  you may just appear down.  it's just no fun
to get the call "i can't (read my mail|cpu in|...)" at 2am.  i'd
much rather the dumb machine panic and reboot automaticly.

but i'm biased because my fileservers doesn't care, and you can't
connect to them remotely.

> not being able to make a
> network connection is a very common case so i guess it would be ok to
> simply let it fail on this point instead of hanging and waiting.
>
> rebooting sounds like a little bit too drastic for me. expecially if it
> is about icmp sockets. devip just has too less context to make a good
> decision.
>
> the comment above the panic call also hinted that it was ment as some
> kind of a bet or a debugging leftover so it felt save to remove it :)

panicing is not right.  all i'm saying is that printing might not really
help either.  i don't think either of us have the testing resources to
be confident that your patch is going to be a significant win.  it
looks fine, but i don't want to find out at 2am it's not really.
after all, i haven't yet had a panic due to this and i've got some
(unintentionally) tough customers.  they've got deadlines, too.

- erik