[9fans] fun with ethernet

[9fans] fun with ethernet

[andrey mirtchovski]

[-- Attachment #1: Type: text/plain, Size: 2901 bytes --]

Hi,

If you seek excitement and thrills you need to look no further than
Plan9 -- it gives you everything and then some, but in a good way (or
at least an interesting way)...

Several months ago we reported problems with TCP/IP over 100baseT and
Gigabit networks -- for 100baseT when we attempted to write data
larger than a certain number (some 130kbytes) to a network
connection's file descriptor it would just stop transmitting, or will
do it very slowly.  This behaviour is fully reproducible with netpipe
[ http://www.acl.lanl.gov/plan9/netpipe/]..  playing with the window
size for ethernet and other niceties got us to increase the largest
amount of data we could send without triggering the bug, but it
ultimately reached a point beyond which the data transmission rate
dropped so significantly, that it was as if no data was going
through...

We ran netpipe on the 4th edition machines (Presotto, I think, has
included some improvements for fast retransmit, etc) and we still
observed the abovementioned drop in bandtwidth.

Due to some lucky circumstances we decided to try iostats(4) and see
where exactly are we losing data (not that iostats is the right tool
for the job, but it's at least entertaining to look at the results).
What was our excitement when we discovered that because of iostats our
latencies were a bit higher, the data transmission rate was a bit
slower, but we were able to transmit _much_ more data than before!

For the first time ever we saw netpipe complete an entire run
(transmitting more than 2gigabytes of data back and forth in the
process -- iostats told us) and see that even with a higher latency we
got better network performance for large data sed than any time
before...

After a bit of testing we discovered that the higher the latency, the
larger the amount of data we're able to transmit before we trigger the
bandtwidth dropout.  I.e. putting 5 switches between the two machines
tested will "fix" the bug.

We haven't made any major attempts to find out where and why this
happens, but preliminary discussions reached the conclusion that we
fill up a queue somewhere in the TCP stack (well, ok, blame me if
that's way off base).

Included are two plots of data produced by netpipe on identical
hardware.  One with no iostats running at all, two with iostats
running on the receive and transmit side respectively and one on both
sides.  One of the plots shows a detail at and around the dropoff
point -- you can see the no-iostats and iostats-on-receive side
dropping after 90k, iostats-on-transmit continues to 131K and drops
afterwards and iostats-on-both goes much, much further.

To test: download netpipe, compile and connect two machines together.
To test with iostats on both sides do:

recv% iostats 8.out -r #the receive side
trmt% iostats 8.ut -t -h <receive address> #transmit side

[-- Attachment #2: iostats.jpg --]
[-- Type: image/jpeg, Size: 27100 bytes --]

[-- Attachment #3: iostats-detail.jpg --]
[-- Type: image/jpeg, Size: 21680 bytes --]