Re: [9fans] tcp problems

Re: [9fans] tcp problems

[presotto]

[-- Attachment #1: Type: text/plain, Size: 894 bytes --]

The -1 means throw away the packet, we've seen it before.  It's
the correct response, tcpiput then ignores the packet (modulo looking
at the ack).

 From your description it sounds like you're going into retransmission
at the send side.  That's because it is timing out waiting for
an ack.  That in turn is either because the acks are not getting
back, getting back too late, or we've got some terrible bug.  We
might have a screwed up timer.

I'ld gotten that far myself before I got interrupted to
justifying my continued existence to my boss.  When one
side starts retransmitting, its congestion window closes
down so things go pretty slow till it grows open again.
If we keep dropping into retransmission mode, things stay
really slow which is what you're seeing.

Keep looking.  Your next step might be to check the acks
arriving at the sender and see if they make sense.

[-- Attachment #2: Type: message/rfc822, Size: 4562 bytes --]

From: andrey mirtchovski <andrey@lanl.gov>
To: 9fans@cse.psu.edu
Subject: [9fans] tcp problems
Date: Fri, 15 Mar 2002 20:54:38 -0700
Message-ID: <20020316035113.A000F19AA1@mail.cse.psu.edu>

Hi,

We've reported before the problems that plan9 is experiencing with tcp 
-- when the data transmitted is more than, let's say,  32Kbytes 
the performance drops somewhat on 100baseT and by a large amount on
Gigabit Ethernet networks.

Dean has been digging in the tcp.c code trying to find what the reason
is and discovered that the amount of packet retransmissions increases
by a factor of 1000+ when the size of the data sent goes above
32kbytes.  Note however that this is not a fixed size that we can
establish scientifically, it's just in the vicinity (playing with the
load on the machine could influence it, sometimes by a large amount).

A kernel profile during execution shows that the machine is idle some
80% of the time during peak transmissions so it is definitely (umm?)
not a hardware issue.


Turning on debugging messages we discovered that the retransmits are
caused by tcptrim() in tcp.c which calculates that a packet is out of
the receive window for tcp and triggers its retransmit.  The actual
code dealing with this is in tcpiput():

	/* Cut the data to fit the receive window */
	if(tcptrim(tcb, &seg, &bp, &length, f) == -1) {
		netlog(f, Logtcp, "tcp len < 0, %lux %d\n", seg.seq,length);
		update(s, &seg);
		if(qlen(s->wq)+tcb->flgcnt == 0 && tcb->state == Closing) {
			netlog(f, Logtcp, "tcp len error halt\n");
			tcphalt(tpriv, &tcb->rtt_timer);
			tcphalt(tpriv, &tcb->acktimer);
			tcphalt(tpriv, &tcb->katimer);
			tcpsetstate(s, Time_wait);
			tcb->timer.start = MSL2*(1000 / MSPTICK);
			tcpgo(tpriv, &tcb->timer);
		}
		if(!(seg.flags & RST)) {
			tcb->flags |= FORCE;
			netlog(f, Logtcp, "tcp len error output\n");
			goto output;
		}
		qunlock(s);
		poperror();
		return;
	}


For the packets we think are causing the problem, tcptrim returns -1,
we log an error message (incorrect, but i'll come to that later), skip
the first 'if' and enter if(!(seg.flags & RST))....

So we did a huge dump of all packets received during a single run of
presotto's netpipe clone (Dean modified it locally so it'd compile)
and found out that the packets that tcptrim rejects are actually not
bad packets, but a repeat of a packet already acknowledged as received
(they fall out of the receive window, but are in front of it..  It
sort of looks like this (I know most of you are familiar with this,
I'm just trying to explain the situation better):

received:		size:		window_start:		window_end:
100			5		100				200
105			5		105				205
110			5		110				210
120			5		115				215	#115 did not arrive orderly
125			5		115				215
130			5		115				215
115			5		115				215	# 115 arrives late
135			5		135				235	
115			5		140				240	# a second 115 arrives retransmitted from the other side


So at this point tcptrim() thinks (correctly) that the packet is out
of the window, returns -1 and the tcp stack prints that the packet
length is negative (incorrectly -- the packet length is reported fine
all the time).  It's a bit of a mistery what the stack decides to do
with the packet after that.  Normally the first thing that comes to
mind is to throw it away, since we're ack-ing a higher sequence
already, but...



Can anyone spot the problem with this code immediately?  Is there a
problem at all?


andrey

Re: [9fans] tcp problems

[plan9]

We also used to test ect2 and elnk3 both wired to an old D-Link so called switch. Auto negociation clearly was a big mess and we were obliged to do it by hand. Further more we discovered that operations became very slOooooow and finaly stated it came from the switch ASIC since a more recent did not produce the same results. The switch got killed, the ploblems (auto-nego / perfs) disapeared. Next time, we may buy a Cisco 19xx or 29xx, those work for sure !

<forsyth@caldo.demon.co.uk> a crit dans le message news:<20020316194329.7904519ABD@mail.cse.psu.edu>...
> i'm sure i saw similar effects when i had a full-duplex link that one side thought
> was half-duplex (or conversely, a half-duplex link that one or both
> treated as full-duplex); i'd always assumed the acks and other packets were lost.
> they could communicate but it got very slow.
> i mention it because hardware-level autoconfiguration between
> cards and switches often got messed up a few years ago.
> i seem to have better luck with things now on 10/100meg, but perhaps there's
> still something not quite right with Gigabit ether kit's negotiation, if you're
> relying on that.  or perhaps the driver isn't setting that quite right.
> probably nothing to do with your problems but i thought i'd mention it just in case.
> 

Re: [9fans] tcp problems

[forsyth]

i'm sure i saw similar effects when i had a full-duplex link that one side thought
was half-duplex (or conversely, a half-duplex link that one or both
treated as full-duplex); i'd always assumed the acks and other packets were lost.
they could communicate but it got very slow.
i mention it because hardware-level autoconfiguration between
cards and switches often got messed up a few years ago.
i seem to have better luck with things now on 10/100meg, but perhaps there's
still something not quite right with Gigabit ether kit's negotiation, if you're
relying on that.  or perhaps the driver isn't setting that quite right.
probably nothing to do with your problems but i thought i'd mention it just in case.

[9fans] tcp problems

[andrey mirtchovski]

Hi,

We've reported before the problems that plan9 is experiencing with tcp 
-- when the data transmitted is more than, let's say,  32Kbytes 
the performance drops somewhat on 100baseT and by a large amount on
Gigabit Ethernet networks.

Dean has been digging in the tcp.c code trying to find what the reason
is and discovered that the amount of packet retransmissions increases
by a factor of 1000+ when the size of the data sent goes above
32kbytes.  Note however that this is not a fixed size that we can
establish scientifically, it's just in the vicinity (playing with the
load on the machine could influence it, sometimes by a large amount).

A kernel profile during execution shows that the machine is idle some
80% of the time during peak transmissions so it is definitely (umm?)
not a hardware issue.


Turning on debugging messages we discovered that the retransmits are
caused by tcptrim() in tcp.c which calculates that a packet is out of
the receive window for tcp and triggers its retransmit.  The actual
code dealing with this is in tcpiput():

	/* Cut the data to fit the receive window */
	if(tcptrim(tcb, &seg, &bp, &length, f) == -1) {
		netlog(f, Logtcp, "tcp len < 0, %lux %d\n", seg.seq,length);
		update(s, &seg);
		if(qlen(s->wq)+tcb->flgcnt == 0 && tcb->state == Closing) {
			netlog(f, Logtcp, "tcp len error halt\n");
			tcphalt(tpriv, &tcb->rtt_timer);
			tcphalt(tpriv, &tcb->acktimer);
			tcphalt(tpriv, &tcb->katimer);
			tcpsetstate(s, Time_wait);
			tcb->timer.start = MSL2*(1000 / MSPTICK);
			tcpgo(tpriv, &tcb->timer);
		}
		if(!(seg.flags & RST)) {
			tcb->flags |= FORCE;
			netlog(f, Logtcp, "tcp len error output\n");
			goto output;
		}
		qunlock(s);
		poperror();
		return;
	}


For the packets we think are causing the problem, tcptrim returns -1,
we log an error message (incorrect, but i'll come to that later), skip
the first 'if' and enter if(!(seg.flags & RST))....

So we did a huge dump of all packets received during a single run of
presotto's netpipe clone (Dean modified it locally so it'd compile)
and found out that the packets that tcptrim rejects are actually not
bad packets, but a repeat of a packet already acknowledged as received
(they fall out of the receive window, but are in front of it..  It
sort of looks like this (I know most of you are familiar with this,
I'm just trying to explain the situation better):

received:		size:		window_start:		window_end:
100			5		100				200
105			5		105				205
110			5		110				210
120			5		115				215	#115 did not arrive orderly
125			5		115				215
130			5		115				215
115			5		115				215	# 115 arrives late
135			5		135				235	
115			5		140				240	# a second 115 arrives retransmitted from the other side


So at this point tcptrim() thinks (correctly) that the packet is out
of the window, returns -1 and the tcp stack prints that the packet
length is negative (incorrectly -- the packet length is reported fine
all the time).  It's a bit of a mistery what the stack decides to do
with the packet after that.  Normally the first thing that comes to
mind is to throw it away, since we're ack-ing a higher sequence
already, but...



Can anyone spot the problem with this code immediately?  Is there a
problem at all?


andrey