Re: [9fans] tcp!

Re: [9fans] tcp!

[Richard Miller]

I reported:

> Within the last month or so I've been having trouble copying large
> files to remote servers e.g. sources.  The cp process hangs for
> many minutes and eventually ends in 'mount rpc error'.

Thanks to a hint from Erik ("... an mss problem of some sort"), I've
managed to make the problem go away, by doing
  echo mtu 1496 >/net/ipifc/1/ctl

I hope to come back to this when I have more time, because I don't
like not understanding why this works.  As nobody else has said they
have the same trouble, there may be something amiss in my adsl gateway.

Re: [9fans] tcp!

[Gorka Guardiola]

I had this problem several years ago
with an adsl router (9fans archive may know about this). There was a bug in my adsl router (which seems to be common, I have seen it since more than once) that dropped ethernet frames of size greater than 1480 (someone counted a header twice probably). Linux adapts the
mss to 1480 if there are problems so it works in this case. 

G.

On Aug 21, 2012, at 8:32 PM, Richard Miller <9fans@hamnavoe.com> wrote:

> I reported:
> 
>> Within the last month or so I've been having trouble copying large
>> files to remote servers e.g. sources.  The cp process hangs for
>> many minutes and eventually ends in 'mount rpc error'.
> 
> Thanks to a hint from Erik ("... an mss problem of some sort"), I've
> managed to make the problem go away, by doing
>  echo mtu 1496 >/net/ipifc/1/ctl
> 
> I hope to come back to this when I have more time, because I don't
> like not understanding why this works.  As nobody else has said they
> have the same trouble, there may be something amiss in my adsl gateway.
> 
> 

Re: [9fans] tcp!

[Steven Stallion]

On Wed, Aug 22, 2012 at 10:18 AM, Gorka Guardiola <paurea@gmail.com> wrote:
> I had this problem several years ago
> with an adsl router (9fans archive may know about this). There was a bug in my adsl router (which seems to be common, I have seen it since more than once) that dropped ethernet frames of size greater than 1480 (someone counted a header twice probably). Linux adapts the
> mss to 1480 if there are problems so it works in this case.

Not so much a bug as ATM overhead.

Re: [9fans] tcp!

[erik quanstrom]

On Wed Aug 22 14:31:13 EDT 2012, sstallion@gmail.com wrote:
> On Wed, Aug 22, 2012 at 10:18 AM, Gorka Guardiola <paurea@gmail.com>
> wrote:
> > I had this problem several years ago with an adsl router (9fans
> > archive may know about this).  There was a bug in my adsl router
> > (which seems to be common, I have seen it since more than once) that
> > dropped ethernet frames of size greater than 1480 (someone counted a
> > header twice probably).  Linux adapts the mss to 1480 if there are
> > problems so it works in this case.
> >
> Not so much a bug as ATM overhead.

atm overhead is 5 bytes per 48 bytes transmitted.

the original problem is a limit of 1496 bytes,
not 1460, which is more constent with mpls
than l2tp (1460) or pppoe (1492).  but all that's
guesswork.

the "bug" here, if there is one, is that there's
neither an icmp message nor fragmentation
nor mss rewriting at the local gateway, which
should (since it's eithernet) not silently drop
mtu-sized frames that it's responsible for
gatewaying.

- erik

[9fans] tcp!

[erik quanstrom]

since it came up, i put my working copy of tcp along with some testing
scripts in /n/sources/contrib/quanstro/tcp.

there are a number of fixes rolled into this, but the main fixes are
- add support for new reno,
- properly handle zero-window probes (on both ends),
- don't confuse the cwind with the receiver's advertized window.  this
particular condition can lead to livelock.
- don't confuse the window scale with the amount of local buffering
we'd like to do.
- and, don't queue tcp infinitely, which can crash kernels.  :-)

i don't have the numbers for the old tcp handy, but i think you'll
be surprised at how much difference there can be.  i saw differences
of 20x when the sender was limited in how fast it could read by the
read rate from user space.

i've included "testscript."  for the two machines i have handy, i get
the following results with new and old tcp.

machine		stack	kernel	0ms delay	1ms delay
ideal		-	386	unlimited	8.19mb/s

xeon x5550	old	386	138mb/s		0.49mb/s  (!)
intel atom	old	386	37.2mb/s		0.10mb/s

amd x4 964	new	386	145mb/s		8.03mb/s
intel e31220	new	amd64	303mb/s		8.15mb/s
intel atom	new	386	67mb/s		8.03mb/s
	# note: i can get up to 80mb/s using forsyth's qmalloc.


- erik

Re: [9fans] tcp!

[erik quanstrom]

> - add support for new reno,

i apoligize for not mentioning that the new reno work
was part of the nix/9k tcp.  i'm not sure who wrote it.

sorry!

also i forgot to mention that this version of qread can
potentially cut the number of reads on tcp channels by up
to 1/2.  one might as well completely satisfy the read,
if possible.  especially since typical iounits (8192) do not
divide up into typical mss-sized (1460) packets evenly.

[...]
	/* if we get here, there's at least one block in the queue */
	if(q->state & Qcoalesce){
		/* when coalescing, 0 length blocks just go away */
		b = q->bfirst;
		if(BLEN(b) <= 0){
			freeb(qremove(q));
			goto again;
		}

		/*
		 * grab the first block and as many following
		 * blocks as will partially fit in the read
		 */
		n = 0;
		l = &first;
		for(;;) {
			*l = qremove(q);
			l = &b->next;
			n += BLEN(b);
			if(n >= len || (b = q->bfirst) == nil)
				break;
		}

- erik

Re: [9fans] tcp!

[Richard Miller]

> also i forgot to mention that this version of qread can
> potentially cut the number of reads on tcp channels by up
> to 1/2.  one might as well completely satisfy the read,
> if possible.

This looks like a good idea for tcp.  But there are other
users of qread, with stricter assumptions.  Aren't you in danger
of breaking the contract of pipe(3) which uses qwrite/qread:

          Writes are atomic up to a certain size, typically 32768
          bytes, that is, each write will be delivered in a single
          read by the recipient, provided the receiving buffer is
          large enough.

To preserve the atomicity of qread/qwrite, maybe tcp should be
coalescing the blocks itself by multiple calls to qread.

Re: [9fans] tcp!

[cinap_lenrek]

its only done on queues that have this flag set i think:

Qcoalesce	= (1<<4),	/* coallesce packets on read */

--
cinap

Re: [9fans] tcp!

[Richard Miller]

> its only done on queues that have this flag set i think:

... and it won't be set for pipes, of course.  Sorry Erik, I should
have studied this more carefully.

I'll try it.

Re: [9fans] tcp!

[erik quanstrom]

> ... and it won't be set for pipes, of course.  Sorry Erik, I should
> have studied this more carefully.
> 
> I'll try it.

no problems.  i'm glad you're double-checking.  nobody i know is immune
from error.  and there's me, myself and i.  so i am 3x as likely to screw up.

i'd be curious to know if this makes a noticable difference on slower machines
like the π with tcptest to self.

- erik

Re: [9fans] tcp!

[erik quanstrom]

> This looks like a good idea for tcp.  But there are other
> users of qread, with stricter assumptions.  Aren't you in danger
> of breaking the contract of pipe(3) which uses qwrite/qread:
>
>           Writes are atomic up to a certain size, typically 32768
>           bytes, that is, each write will be delivered in a single
>           read by the recipient, provided the receiving buffer is
>           large enough.

this change only applies to Qcoalesce queues.

the only users of Qcoalesce are the kprintoq and tcp.  both
should be okay with this change.

; g qopen port/devpipe.c
port/devpipe.c:68: 	p->q[0] = qopen(conf.pipeqsize, 0, 0, 0);
port/devpipe.c:73: 	p->q[1] = qopen(conf.pipeqsize, 0, 0, 0);

- erik

Re: [9fans] tcp!

[Richard Miller]

Within the last month or so I've been having trouble copying large
files to remote servers e.g. sources.  The cp process hangs for
many minutes and eventually ends in 'mount rpc error'.  I was
hoping this tcp patch might solve it, but alas no.

Has anyone else been observing this?

Re: [9fans] tcp!

[erik quanstrom]

On Sun Aug 19 09:19:23 EDT 2012, 9fans@hamnavoe.com wrote:
> Within the last month or so I've been having trouble copying large
> files to remote servers e.g. sources.  The cp process hangs for
> many minutes and eventually ends in 'mount rpc error'.  I was
> hoping this tcp patch might solve it, but alas no.

could you send a snoopy capture?  -M100 and just the tail should
be good enough.  also a capture of /net/log with 'set tcp' during the
issue could be helpful. also, could you point to a particular large
file on sources?  i'd like to try to replicate.

- erik