Re: [9fans] fs performance

[Bakul Shah <bakul+plan9@bitblocks.com>]

On Sun, 09 Jan 2011 09:29:04 PST ron minnich <rminnich@gmail.com>  wrote:
>
> Those are interesting numbers. Actually, however, changing a program
> to use the stream stuff is trivial. I would expect the streaming to be
> a real loser in a site with 10GE but we can try it. As John has
> pointed out the streaming only makes sense where the inherent network
> latency is pretty high (10s of milliseconds), i.e. the wide area.

For "streaming" at wirespeed you want to keep the pipe full
at all times.  For that to happen, by the time ack for byte#
N arrives, you would have sent at least RTT*bandwidth more
bytes. So it is the latency*bandwidth product and not just
latency that matters -- that is, streaming makes sense at
lower latencies for higher speed networks.

Ideally on a 10GbE you should get something like 1.1GBps of
TCP bandwidth.  If the throughput you get from local
filesystems is not in this neighborhood, that would mean
there are bigger bottlenecks than dealing with latency.  In
this case streaming won't help much.

On FreeBSD (on a 3.2Ghz w3550) dd </dev/md0 > /dev/null
yields 3.5GBps (@128k blocksize).  md0 is a ramdisk.  Write
to is at about 1/4th read speed.  dd </dev/zero >/dev/null
gives 15+GBps -- basically system overhead for IO. dd from an
uncached file on ZFS is about 350MBps. Once the file is
cached, the throughput increases to 2.5GBps. None of these
use any streaming (though there *is* readahead at the FS
level).

[GBps == 10^9 bytes/second. MBps == 1^6 bytes/second]

The point of mentioning FreeBSD numbers is to show what is
possible. To really improve plan9 fs performance one would
have to look at things like syscall overhead, number of data
copies made, number of syscalls and context switches etc. and
tune each component.

[Aside: has ttcp been ported to plan9?]