Re: [9fans] soverflow for fx->in

Re: [9fans] soverflow for fx->in

[erik quanstrom]

On Mon Oct 26 12:35:07 EDT 2009, andreer@pvv.ntnu.no wrote:
> Our cpu/auth/file went belly up some time during the weekend. It
> responded to ping, but we couldn't drawterm in. No response from
> the vblade process providing storage to our XenServers either.
> Walking down to the machine I found the screen full of "soverflow
> for fx->in" messages. I had to reboot it to get the virtual machines
> back online, so this is my only clue (print statement in
> /sys/src/9/pc/devether.c:186 ). The machine had been working happily
> for over a week before this. Any ideas as to how i can keep this
> from happening again?
>
> The ethernet controller is an "Intel®PRO/1000 GT Desktop Adapter"
> according to the sticker on the back, on bootup i get
> #l0: i82543: 1000 Mbps port 0xFEBE0000 irq 5 tu 1514:000e0c73d17a

a soft overflow is when there are packets coming in
that don't fit on the ethernet queue.  soft overflows
are not bugs.  it's a consequence of having a finite
memory; one cannot queue infinitely.

however, printing each soft overflow to the console
can put one in a difficult to recover from situation.
i would suggest reverting these changes

; diff -c /n/sourcesdump/2009/0601/plan9/sys/src/9/pc/devether.c /n/sourcesdump/2009/0404/plan9/sys/src/9/pc/devether.c
/n/sourcesdump/2009/0601/plan9/sys/src/9/pc/devether.c:182,196 - /n/sourcesdump/2009/0404/plan9/sys/src/9/pc/devether.c:182,192
  				else if(xbp = iallocb(len)){
  					memmove(xbp->wp, pkt, len);
  					xbp->wp += len;
- 					if(qpass(f->in, xbp) < 0) {
- 						print("soverflow for f->in\n");
+ 					if(qpass(f->in, xbp) < 0)
  						ether->soverflows++;
- 					}
  				}
- 				else {
- 					print("soverflow iallocb\n");
+ 				else
  					ether->soverflows++;
- 				}
  			}
  			else
  				etherrtrace(f, pkt, len);
/n/sourcesdump/2009/0601/plan9/sys/src/9/pc/devether.c:198,207 - /n/sourcesdump/2009/0404/plan9/sys/src/9/pc/devether.c:194,201
  	}

  	if(fx){
- 		if(qpass(fx->in, bp) < 0) {
- 			print("soverflow for fx->in\n");
+ 		if(qpass(fx->in, bp) < 0)
  			ether->soverflows++;
- 		}
  		return 0;
  	}
  	if(fromwire){
/n/sourcesdump/2009/0601/plan9/sys/src/9/pc/devether.c:239,246 - /n/sourcesdump/2009/0404/plan9/sys/src/9/pc/devether.c:233,238
  	}

  	if(!loopback){
- 		if(qfull(ether->oq))
- 			print("etheroq: WARNING: ether->oq full!\n");
  		qbwrite(ether->oq, bp);
  		if(ether->transmit != nil)
  			ether->transmit(ether);


- erik

[9fans] soverflow for fx->in

[James Chapman]

On Sat, Jul 24, 2010 at 10:44 PM, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
>> what's happening is that the printing of "soverflow for fx->in"
>> is wasting so much time that the imap server gets bored and hangs
>> up.
>
> This is really dragging out for 30 minutes?

No. Not that long.

Also, I just did a pull from sources which appears to have updated
almost the complete contents of /386. This caused the "soverflow for
fx->in" message to appear. I had to run pull three times to get it to
finish.

James

Re: [9fans] soverflow for fx->in

[erik quanstrom]

> Also, I just did a pull from sources which appears to have updated
> almost the complete contents of /386. This caused the "soverflow for
> fx->in" message to appear. I had to run pull three times to get it to
> finish.

maybe my first message wasn't that clear.  any ethernet activity
can cause this if the input queue can't be drained fast enough.
clearly, too much of the cpu is spent in interrupt context receiving
packets, and not enough processing them. but delete the print, and
tcp congestion avoidance should make up the difference fast enough
to keep the remote happy.

- erik

Re: [9fans] soverflow for fx->in

[James Chapman]

Thanks Erik. Removing the first instance of the print in
/sys/src/9/pc/devether.c appears to fix it:

/* print("soverflow for f->in\n");  */

There is another another print lower down but I left this there for the moment.

term% time upas/fs -f /imaps/imap.gmail.com/address_removed@gmail.com

!Adding key: proto=pass server=imap.gmail.com service=imap
user=address_removed@gmail.com
password:
!
0.21u 1.04s 35.59r 	 upas/fs -f /imaps/imap.gmail.com/address_removed@gmail.com

This was for 595 messages actually.

Many thanks,

James

On Sun, Jul 25, 2010 at 7:52 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> Also, I just did a pull from sources which appears to have updated
>> almost the complete contents of /386. This caused the "soverflow for
>> fx->in" message to appear. I had to run pull three times to get it to
>> finish.
>
> maybe my first message wasn't that clear.  any ethernet activity
> can cause this if the input queue can't be drained fast enough.
> clearly, too much of the cpu is spent in interrupt context receiving
> packets, and not enough processing them. but delete the print, and
> tcp congestion avoidance should make up the difference fast enough
> to keep the remote happy.
>
> - erik
>
>

[9fans] soverflow for fx->in

[Andreas Eriksen]

Our cpu/auth/file went belly up some time during the weekend. It
responded to ping, but we couldn't drawterm in. No response from
the vblade process providing storage to our XenServers either.
Walking down to the machine I found the screen full of "soverflow
for fx->in" messages. I had to reboot it to get the virtual machines
back online, so this is my only clue (print statement in
/sys/src/9/pc/devether.c:186 ). The machine had been working happily
for over a week before this. Any ideas as to how i can keep this
from happening again?

The ethernet controller is an "Intel®PRO/1000 GT Desktop Adapter"
according to the sticker on the back, on bootup i get
#l0: i82543: 1000 Mbps port 0xFEBE0000 irq 5 tu 1514:000e0c73d17a