From mboxrd@z Thu Jan 1 00:00:00 1970 From: erik quanstrom Date: Thu, 29 Apr 2010 13:23:00 -0400 To: 9fans@9fans.net Message-ID: In-Reply-To: <20100429170859.2EBE05B78@mail.bitblocks.com> References: <5fa9fbfe115a9cd5a81d0feefe413192@quintile.net> <4fa1305e0f56a0ef89c2e05320fa5997@coraid.com> <40cf59cfc2735e232f0fd67df725e65d@kw.quanstro.net> <046b9c874815d108c6ffb7a857e847a7@kw.quanstro.net> <20100429170859.2EBE05B78@mail.bitblocks.com> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Subject: Re: [9fans] A simple experiment Topicbox-Message-UUID: 12d31422-ead6-11e9-9d60-3106f5b1d025 > > 9p, like aoe, is a ping-pong protocol. each message requires an ack. > > therefore, the transport layer doesn't need flow control. > > Therefore, it is also not able to utilise bandwidth > effectively over longhaul links. As an example, US coast > to coast round trip time latency is about 100ms. Now consider > fcp. Each worker thread of fcp does 8K read/writes. Due to > pingponging, the *most* a thread can xfer coast to coast is > 80KBps (for 16 threads, 1.28MBps). It is actually much worse > since each thread doesn't even overlap reads with writes. i think you are conflating a ping-pong protocol with the limitation of a single outstanding message. there is no reason that one needs a 1:1 correspondence between threads and outstanding messages. nor does the protocol inherently prohibit 1 write from becoming multiple messages. in the case of aoe, you can have up to 2^32 - 1 outstanding packets at the same time to each target. thus the theoretical maximum would be 80 * (2^31 - 1) kbps per target. the aoe driver (in contrib) has a limit of 128 8k jumbo frames outstanding per target. the aoe driver has a fixed number of thread per target. > Short of a sliding window that is as large as the capacity of > the pipe, you can't expect to keep it full. As usual one has > to trade off simplicity vs performance. i don't think so. the only thing a sliding window gives you is < 1 ack per message sent. -erik