[9fans] A simple experiment

[9fans] A simple experiment

[ron minnich]

I had interest in being able to see plan 9 source at bitbucket.org.
Part of the driver was my continuing inability to get replica to work
well at home, and part just a need to tinker :-)

So, I created an empty repo at bitbucket.org,
http://bitbucket.org/rminnich/sysfromiso/overview

and then did the usual
hg clone -e '/bin/openssh/ssh -2' ssh://hg@bitbucket.org/rminnich/sysfromiso

At this point on Plan 9 I have a directory, sysfromiso, that is empty
save for a .hg

Now on linux or other systems, you copy a bunch of directories in
there, hg add them, and away you go.

Plan 9 is more interesting:

hget http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
 rc -c 'cd /tmp; bunzip2 iso.bz2'
 9660srv -f /tmp/iso iso
mount /srv/iso /n/iso

now I've got the sources over there in /n/iso. What's next?

Simple:

cd sysfromiso
bind -a /n/iso .

And then add some trees:
hg add sys/src

then
hg commit
hg push -e '/bin/openssh/ssh -2'

And I've got a starting point. What's interesting is that the
directory always looks empty until I do the bind:
term% ls sysfromiso
sysfromiso/.hg
term%

So the script to continue updating the repo is pretty simple:
 #!/bin/rc
hget http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
 rc -c 'cd /tmp; bunzip2 iso.bz2'
 9660srv -f /tmp/iso iso
mount /srv/iso /n/iso
ape/psh
cd sysfromiso
bind -b /n/iso .
x=`date`
hg commit -m "$x"
hg push -e  '/bin/openssh/ssh -2'

(note I need ape/psh when I use ssh for pushes -- quoting rules issue)

This can be run from cron -- once you get through the ssh issues I
mentioned in the earlier note.

Result is an hg repo on bitbucket.org that I can get to from anywhere,
and I can watch as Geoff continues to beat on the kw port :-)

More importantly, it's going to be easier for me to bisect and find
problems when I build from kernel source, which is very handy in my
case. The web interface of bitbucket gives me a pretty reasonable way
to compare different revs. I'm offering this note in the event others
want to use this interface and repo.

ron

Re: [9fans] A simple experiment

[erik quanstrom]

> More importantly, it's going to be easier for me to bisect and find
> problems when I build from kernel source, which is very handy in my
> case. The web interface of bitbucket gives me a pretty reasonable way
> to compare different revs. I'm offering this note in the event others
> want to use this interface and repo.

is plan 9 really that complicated that bisect is a useful tool?
perhaps i'm still in the dark ages, but i've done fine with
diff(1) and history(1).

perhaps the key difference is that i sync with sources by
diffing and merging by hand.  /n/sources/lsr is a great guide
to Things That Have Changed.  pull is a little too scary if you're
responsible for keeping things running.

- erik

Re: [9fans] A simple experiment

[jake]

Nice work, but couldn't you just bind /n/sources/plan9/sys/src
to the hg repo and push from there?

Re: [9fans] A simple experiment

[John Floren]

On Tue, Apr 27, 2010 at 1:54 PM,  <jake@9srv.net> wrote:
> Nice work, but couldn't you just bind /n/sources/plan9/sys/src
> to the hg repo and push from there?
>

That would almost certainly be slower than grabbing the ISO via HTTP
and getting the file tree locally.


John

Re: [9fans] A simple experiment

[erik quanstrom]

On Tue Apr 27 13:58:39 EDT 2010, slawmaster@gmail.com wrote:
> On Tue, Apr 27, 2010 at 1:54 PM,  <jake@9srv.net> wrote:
> > Nice work, but couldn't you just bind /n/sources/plan9/sys/src
> > to the hg repo and push from there?
> >
>
> That would almost certainly be slower than grabbing the ISO via HTTP
> and getting the file tree locally.

it would be interesting to try.  if hg can push in parallel, it
could be competitive.  fetching the iso, decompressing the iso,
etc are not free.  and you can't push anything until after step
2.  talk about killer latency.

- erik

Re: [9fans] A simple experiment

[Francisco J Ballesteros]

FWIW, I pull -n first, then see what´s going to change, then
merge what I care about, then pull without -n, adding -s´s and -c´s as
a result of which file I merged, and I´m done.

Regarding pulls from sources I´ve found no problem so far (other than speed).
Indeed, speed is not a problem here because we pull from a local
mirror of replica
(also updated via replica).

Internally, I´ve switched from replica to repl (similar to replica,
but a custom tool)
only because it´s easier to sync trees that I know are in sync (e.g.,
because I copied
files by hand).

hth

On Tue, Apr 27, 2010 at 7:49 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> pull is a little too scary if you're
> responsible for keeping things running.
>
> - erik
>
>

Re: [9fans] A simple experiment

[John Floren]

On Tue, Apr 27, 2010 at 1:59 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> On Tue Apr 27 13:58:39 EDT 2010, slawmaster@gmail.com wrote:
>> On Tue, Apr 27, 2010 at 1:54 PM,  <jake@9srv.net> wrote:
>> > Nice work, but couldn't you just bind /n/sources/plan9/sys/src
>> > to the hg repo and push from there?
>> >
>>
>> That would almost certainly be slower than grabbing the ISO via HTTP
>> and getting the file tree locally.
>
> it would be interesting to try.  if hg can push in parallel, it
> could be competitive.  fetching the iso, decompressing the iso,
> etc are not free.  and you can't push anything until after step
> 2.  talk about killer latency.
>
> - erik
>
>

My experiments have shown that copying a large file via HTTP is
significantly faster than copying the same file via 9P. I haven't
tested it, but I would wager that opening, reading, and closing
hundreds of small files via 9P would also be much slower than grabbing
the same files in a compressed archive via HTTP and uncompressing.
Also, I'm not sure of the exact mechanics of hg, but I'm guessing that
a commit + push would involve at least two traversals of the tree,
which is not fun when you're hitting sources for every file op.


John

Re: [9fans] A simple experiment

[Jorden M]

On Tue, Apr 27, 2010 at 2:35 PM, John Floren <slawmaster@gmail.com> wrote:
> On Tue, Apr 27, 2010 at 1:59 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> On Tue Apr 27 13:58:39 EDT 2010, slawmaster@gmail.com wrote:
>>> On Tue, Apr 27, 2010 at 1:54 PM,  <jake@9srv.net> wrote:
>>> > Nice work, but couldn't you just bind /n/sources/plan9/sys/src
>>> > to the hg repo and push from there?
>>> >
>>>
>>> That would almost certainly be slower than grabbing the ISO via HTTP
>>> and getting the file tree locally.
>>
>> it would be interesting to try.  if hg can push in parallel, it
>> could be competitive.  fetching the iso, decompressing the iso,
>> etc are not free.  and you can't push anything until after step
>> 2.  talk about killer latency.
>>
>> - erik
>>
>>
>
> My experiments have shown that copying a large file via HTTP is
> significantly faster than copying the same file via 9P. I haven't
> tested it, but I would wager that opening, reading, and closing
> hundreds of small files via 9P would also be much slower than grabbing
> the same files in a compressed archive via HTTP and uncompressing.
> Also, I'm not sure of the exact mechanics of hg, but I'm guessing that
> a commit + push would involve at least two traversals of the tree,
> which is not fun when you're hitting sources for every file op.
>
>
> John
>
>

Assuming hg acts like most other DVCS:

The commit would happen locally to the `database' in the .hg
directory. The sources tree would be read once to do the commit into
blobs in .hg. Push would then only send stuff from that .hg directory.

Most DVCS don't touch the working directory for push/pull operations,
they simply sync up the .hg/.git/.foo directories. Merging and such
are all done after that sync, locally.

Re: [9fans] A simple experiment

[Skip Tavakkolian]

> My experiments have shown that copying a large file via HTTP is
> significantly faster than copying the same file via 9P.

were you using fcp?

i'm curious as to where the differences could come from, since the
usual suspects that can make the difference (establishing a
connection, sequential reads, directory walks/stats) wont come into
play for one file.

Re: [9fans] A simple experiment

[Federico G. Benavento]

looks like you got it going

fetching http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2
would improve your dowload speed as it only contains the
source

On Tue, Apr 27, 2010 at 2:38 PM, ron minnich <rminnich@gmail.com> wrote:
> I had interest in being able to see plan 9 source at bitbucket.org.
> Part of the driver was my continuing inability to get replica to work
> well at home, and part just a need to tinker :-)
>
> So, I created an empty repo at bitbucket.org,
> http://bitbucket.org/rminnich/sysfromiso/overview
>
> and then did the usual
> hg clone -e '/bin/openssh/ssh -2' ssh://hg@bitbucket.org/rminnich/sysfromiso
>
> At this point on Plan 9 I have a directory, sysfromiso, that is empty
> save for a .hg
>
> Now on linux or other systems, you copy a bunch of directories in
> there, hg add them, and away you go.
>
> Plan 9 is more interesting:
>
> hget http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
>  rc -c 'cd /tmp; bunzip2 iso.bz2'
>  9660srv -f /tmp/iso iso
> mount /srv/iso /n/iso
>
> now I've got the sources over there in /n/iso. What's next?
>
> Simple:
>
> cd sysfromiso
> bind -a /n/iso .
>
> And then add some trees:
> hg add sys/src
>
> then
> hg commit
> hg push -e '/bin/openssh/ssh -2'
>
> And I've got a starting point. What's interesting is that the
> directory always looks empty until I do the bind:
> term% ls sysfromiso
> sysfromiso/.hg
> term%
>
> So the script to continue updating the repo is pretty simple:
>  #!/bin/rc
> hget http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
>  rc -c 'cd /tmp; bunzip2 iso.bz2'
>  9660srv -f /tmp/iso iso
> mount /srv/iso /n/iso
> ape/psh
> cd sysfromiso
> bind -b /n/iso .
> x=`date`
> hg commit -m "$x"
> hg push -e  '/bin/openssh/ssh -2'
>
> (note I need ape/psh when I use ssh for pushes -- quoting rules issue)
>
> This can be run from cron -- once you get through the ssh issues I
> mentioned in the earlier note.
>
> Result is an hg repo on bitbucket.org that I can get to from anywhere,
> and I can watch as Geoff continues to beat on the kw port :-)
>
> More importantly, it's going to be easier for me to bisect and find
> problems when I build from kernel source, which is very handy in my
> case. The web interface of bitbucket gives me a pretty reasonable way
> to compare different revs. I'm offering this note in the event others
> want to use this interface and repo.
>
> ron
>
>



-- 
Federico G. Benavento

Re: [9fans] A simple experiment

[EBo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2783 bytes --]

"Federico G. Benavento" <benavento@gmail.com> said:

> looks like you got it going
>
> fetching http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2
> would improve your dowload speed as it only contains the
> source
>
> On Tue, Apr 27, 2010 at 2:38 PM, ron minnich <rminnich@gmail.com> wrote:
> > I had interest in being able to see plan 9 source at bitbucket.org.
> > Part of the driver was my continuing inability to get replica to work
> > well at home, and part just a need to tinker :-)
> >
> > So, I created an empty repo at bitbucket.org,
> > http://bitbucket.org/rminnich/sysfromiso/overview
> >
> > and then did the usual
> > hg clone -e '/bin/openssh/ssh -2' ssh://hg@bitbucket.org/rminnich/sysfromiso
> >
> > At this point on Plan 9 I have a directory, sysfromiso, that is empty
> > save for a .hg
> >
> > Now on linux or other systems, you copy a bunch of directories in
> > there, hg add them, and away you go.
> >
> > Plan 9 is more interesting:
> >
> > hget http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
> >  rc -c 'cd /tmp; bunzip2 iso.bz2'
> >  9660srv -f /tmp/iso iso
> > mount /srv/iso /n/iso
> >
> > now I've got the sources over there in /n/iso. What's next?
> >
> > Simple:
> >
> > cd sysfromiso
> > bind -a /n/iso .
> >
> > And then add some trees:
> > hg add sys/src
> >
> > then
> > hg commit
> > hg push -e '/bin/openssh/ssh -2'
> >
> > And I've got a starting point. What's interesting is that the
> > directory always looks empty until I do the bind:
> > term% ls sysfromiso
> > sysfromiso/.hg
> > term%
> >
> > So the script to continue updating the repo is pretty simple:
> >  #!/bin/rc
> > hget http://plan9.bell-labs.com/plan9/download/plan9.iso.bz2>/tmp/iso.bz2
> >  rc -c 'cd /tmp; bunzip2 iso.bz2'
> >  9660srv -f /tmp/iso iso
> > mount /srv/iso /n/iso
> > ape/psh
> > cd sysfromiso
> > bind -b /n/iso .
> > x=`date`
> > hg commit -m "$x"
> > hg push -e  '/bin/openssh/ssh -2'
> >
> > (note I need ape/psh when I use ssh for pushes -- quoting rules issue)
> >
> > This can be run from cron -- once you get through the ssh issues I
> > mentioned in the earlier note.
> >
> > Result is an hg repo on bitbucket.org that I can get to from anywhere,
> > and I can watch as Geoff continues to beat on the kw port :-)
> >
> > More importantly, it's going to be easier for me to bisect and find
> > problems when I build from kernel source, which is very handy in my
> > case. The web interface of bitbucket gives me a pretty reasonable way
> > to compare different revs. I'm offering this note in the event others
> > want to use this interface and repo.
> >
> > ron
> >
> >
>
>
>
> --
> Federico G. Benavento
>
>



--

Re: [9fans] A simple experiment

[ron minnich]

On Tue, Apr 27, 2010 at 10:49 AM, erik quanstrom <quanstro@quanstro.net> wrote:

> is plan 9 really that complicated that bisect is a useful tool?
> perhaps i'm still in the dark ages, but i've done fine with
> diff(1) and history(1).

Not good enough for me.

>
> perhaps the key difference is that i sync with sources by
> diffing and merging by hand.  /n/sources/lsr is a great guide
> to Things That Have Changed.  pull is a little too scary if you're
> responsible for keeping things running.

I find that something like hg is just way easier to deal with than the
standard plan 9 tools.

ron

Re: [9fans] A simple experiment

[ron minnich]

On Tue, Apr 27, 2010 at 10:59 AM, erik quanstrom <quanstro@quanstro.net> wrote:

> it would be interesting to try.  if hg can push in parallel, it
> could be competitive.  fetching the iso, decompressing the iso,
> etc are not free.  and you can't push anything until after step
> 2.  talk about killer latency.

pulling and decompressing the iso are far, far faster for me than
replica. There's no comparison.

ron

Re: [9fans] A simple experiment

[erik quanstrom]

> > is plan 9 really that complicated that bisect is a useful tool?
> > perhaps i'm still in the dark ages, but i've done fine with
> > diff(1) and history(1).
>
> Not good enough for me.

why?

- erik

Re: [9fans] A simple experiment

[ron minnich]

On Tue, Apr 27, 2010 at 12:03 PM, Skip Tavakkolian <9nut@9netics.com> wrote:
>> My experiments have shown that copying a large file via HTTP is
>> significantly faster than copying the same file via 9P.
>
> were you using fcp?
>
> i'm curious as to where the differences could come from, since the
> usual suspects that can make the difference (establishing a
> connection, sequential reads, directory walks/stats) wont come into
> play for one file.
>

fcp is still a 9p conversation. http get is a tcp stream. Fcp is
better than cp but not that much better.

If you're yanking one file, a TCP stream is pretty ideal. Dropping 9p
on top of it, even when the 9p involves multiple TREADs
in flight, is just making things slower.

ron

Re: [9fans] A simple experiment

[Steve Simon]

Ok, I admit its a trivial experiment but:

> fcp is still a 9p conversation. http get is a tcp stream. Fcp is
> better than cp but not that much better.

> If you're yanking one file, a TCP stream is pretty ideal. Dropping 9p
> on top of it, even when the 9p involves multiple TREADs
> in flight, is just making things slower.

	larch% time cp /n/sources/extra/plan9.tar.bz2 /dev/null
	0.02u 0.52s 647.90r 	 cp /n/sources/extra/plan9.tar.bz2 /dev/null

	larch% time fcp /n/sources/extra/plan9.tar.bz2 /dev/null
	0.01u 0.85s 49.69r 	 fcp /n/sources/extra/plan9.tar.bz2 /dev/null

	larch% time hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2 > /dev/null
	0.37u 0.54s 32.84r 	 hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2

Mmm, HTTP does give better performance, but its not that extreme,
and for a nightly cron script I would not worry about it.
I admit I am surprised by how much a difference there is, it should
be just Tread and Rread headers shouldn't it?

-Steve

Re: [9fans] A simple experiment

[Ethan Grammatikidis]

On 28 Apr 2010, at 12:51, Steve Simon wrote:

> Ok, I admit its a trivial experiment but:
>
>> fcp is still a 9p conversation. http get is a tcp stream. Fcp is
>> better than cp but not that much better.
>
>> If you're yanking one file, a TCP stream is pretty ideal. Dropping 9p
>> on top of it, even when the 9p involves multiple TREADs
>> in flight, is just making things slower.
>
> 	larch% time cp /n/sources/extra/plan9.tar.bz2 /dev/null
> 	0.02u 0.52s 647.90r 	 cp /n/sources/extra/plan9.tar.bz2 /dev/null
>
> 	larch% time fcp /n/sources/extra/plan9.tar.bz2 /dev/null
> 	0.01u 0.85s 49.69r 	 fcp /n/sources/extra/plan9.tar.bz2 /dev/null
>
> 	larch% time hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2
>  > /dev/null
> 	0.37u 0.54s 32.84r 	 hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2
>
> Mmm, HTTP does give better performance, but its not that extreme,
> and for a nightly cron script I would not worry about it.
> I admit I am surprised by how much a difference there is, it should
> be just Tread and Rread headers shouldn't it?

Could round-trip times be adding up? Does 9p do one file at once
strictly?

>
> -Steve
>

--
Simplicity does not precede complexity, but follows it. -- Alan Perlis

Re: [9fans] A simple experiment

[erik quanstrom]

> 	larch% time cp /n/sources/extra/plan9.tar.bz2 /dev/null
> 	0.02u 0.52s 647.90r 	 cp /n/sources/extra/plan9.tar.bz2 /dev/null
>
> 	larch% time fcp /n/sources/extra/plan9.tar.bz2 /dev/null
> 	0.01u 0.85s 49.69r 	 fcp /n/sources/extra/plan9.tar.bz2 /dev/null
>
> 	larch% time hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2 > /dev/null
> 	0.37u 0.54s 32.84r 	 hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2
>
> Mmm, HTTP does give better performance, but its not that extreme,
> and for a nightly cron script I would not worry about it.
> I admit I am surprised by how much a difference there is, it should
> be just Tread and Rread headers shouldn't it?

local weather conditions?  i get a much tighter grouping and fcp
is faster than http:

minooka; time cp /n/sources/extra/plan9.tar.bz2 /dev/null
0.00u 0.18s 343.83r 	 cp /n/sources/extra/plan9.tar.bz2 /dev/null
minooka; time fcp /n/sources/extra/plan9.tar.bz2 /dev/null
0.00u 0.20s 54.64r 	 fcp /n/sources/extra/plan9.tar.bz2 /dev/null
minooka; time hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2 >/dev/null
0.20u 0.20s 55.05r 	 hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2

and for good measure:
minooka; time hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2 > /n/other/quanstro/plan9.tar.bz2
0.17u 0.89s 55.37r 	 hget http://plan9.bell-labs.com/sources/extra/plan9.tar.bz2

if i do a different test where the remote as 1mb of bandwidth
(43ms ping):

ladd; time cp /n/t1/n/other/quanstro/plan9.tar.bz2 /dev/null
0.00u 0.19s 302.93r 	 cp /n/t1/n/other/quanstro/plan9.tar.bz2 /dev/null
ladd; time fcp /n/t1/n/other/quanstro/plan9.tar.bz2 /dev/null
0.00u 0.25s 179.87r 	 fcp /n/t1/n/other/quanstro/plan9.tar.bz2 /dev/null
ladd; time hget http://t1.example.com/files/extra/plan9.tar.bz2 > /dev/null
0.20u 0.24s 175.94r 	 hget http://t1.example.com/files/extra/plan9.tar.bz2

the original test i did was looked funny because i'd used ssl for
the import but regular http.  so the system time was skewed

i wonder if some of the really bad 9p performance could be
related to aan.

- erik

Re: [9fans] A simple experiment

[erik quanstrom]

> Could round-trip times be adding up? Does 9p do one file at once
> strictly?

i think you mean, does 9p have 1 outstanding message.
it does not.  9p supports having as many outstanding
messages as one wishes.  if you do a pread(2), the kernel
will only maintain a single outstanding message for you.
fcp is a quick hack around this.  by keeping n threads
each doing a read, you can keep n messages outstanding.

there is nothing (save complexity) preventing the kernel
from keeping multiple outstanding 9p messages.

this is the technique aoe uses to overcome drive latency,
which — even for enterprise slc ssds — can be considerable
in the worst case (~500ms).

- erik

Re: [9fans] A simple experiment

[Ethan Grammatikidis]

On 28 Apr 2010, at 14:26, erik quanstrom wrote:

>> Could round-trip times be adding up? Does 9p do one file at once
>> strictly?
>
> i think you mean, does 9p have 1 outstanding message.
> it does not.  9p supports having as many outstanding
> messages as one wishes.  if you do a pread(2), the kernel
> will only maintain a single outstanding message for you.
> fcp is a quick hack around this.  by keeping n threads
> each doing a read, you can keep n messages outstanding.
>
> there is nothing (save complexity) preventing the kernel
> from keeping multiple outstanding 9p messages.
>
> this is the technique aoe uses to overcome drive latency,
> which — even for enterprise slc ssds — can be considerable
> in the worst case (~500ms).
>
> - erik
>

Yeah, if I understand right that is what I meant. Good to know, but  
I'm back to the drawing board on why 9p is sometimes so very much  
slower than transferring the equivalent files in a tar file over http.  
Maybe fcp should be used with a large number of threads? :s

-- 
Simplicity does not precede complexity, but follows it. -- Alan Perlis

Re: [9fans] A simple experiment

[erik quanstrom]

> Yeah, if I understand right that is what I meant. Good to know, but
> I'm back to the drawing board on why 9p is sometimes so very much
> slower than transferring the equivalent files in a tar file over http.

rtt.

> Maybe fcp should be used with a large number of threads? :s

the only limitation of the fcp approach is that it is built on a reliable
protocol, so it's impossible to control the number of inflight requests
based on missed packets.

perhaps the real key to a 9p2015 that solves this problem would be
building it on unreliable transport.

by the way, i was trying to generate aan numbers.  but fcp
just breaks aan.  aan is currently transferring about 10bytes/sec.

- erik

Re: [9fans] A simple experiment

[Charles Forsyth]

when timing http it's also a good idea to check that your
isp hasn't got the item cached. admittedly, that in itself
is a further potential advantage to http, because http
caches abound, whereas 9p-based caches are few and far between.

Re: [9fans] A simple experiment

[Tim Newsham]

> I admit I am surprised by how much a difference there is, it should
> be just Tread and Rread headers shouldn't it?

If you have high latency or high bandwidth, the maximum message
size for the 9p messages will be too small to keep the pipe full
if you're using read serially. Did you take a look at how much
bandwith was actually in use during your tests?

> -Steve

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

Re: [9fans] A simple experiment

[Tim Newsham]

My apologies, I didn't notice you had used fcp as well.  Oops.

On Wed, 28 Apr 2010, Tim Newsham wrote:

>> I admit I am surprised by how much a difference there is, it should
>> be just Tread and Rread headers shouldn't it?
>
> If you have high latency or high bandwidth, the maximum message
> size for the 9p messages will be too small to keep the pipe full
> if you're using read serially. Did you take a look at how much
> bandwith was actually in use during your tests?
>
>> -Steve
>
> Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com
>
>

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

Re: [9fans] A simple experiment

[erik quanstrom]

On Wed Apr 28 13:58:40 EDT 2010, newsham@lava.net wrote:
> > I admit I am surprised by how much a difference there is, it should
> > be just Tread and Rread headers shouldn't it?
>
> If you have high latency or high bandwidth, the maximum message
> size for the 9p messages will be too small to keep the pipe full
> if you're using read serially. Did you take a look at how much
> bandwith was actually in use during your tests?

unless you've got some sort of interrupt or cpu bottleneck,
i don't see what message size has to do with this issue.  the
real issue is the # of messages outstanding.

- erik

Re: [9fans] A simple experiment

[ron minnich]

On Wed, Apr 28, 2010 at 11:00 AM, erik quanstrom
<quanstro@labs.coraid.com> wrote:

> unless you've got some sort of interrupt or cpu bottleneck,
> i don't see what message size has to do with this issue.  the
> real issue is the # of messages outstanding.

Trivially, it always matters: what if msize is 1?

There's always a balance between packet size and  parallelism in the #
ops you send (i.e. outstanding packets in flight) and other factors I
forget right now :-)

Too big messages cause heartbreak and bad behavior in some networks.
Too many messages in flight cause issues at each end.

And, in the end, if what you want is a stream, then discrete
request/response transactions can be made to work, but they don't
always work as well.

We did a simple experiment recently: added a new 9p type called
Tstream, because this issue of streams vs. transactions has been
bugging me for years. The semantics are simple: it's a lot like Tread
(almost same packet) but a single Tstream results in any number of
Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
(/usr/rminnich/movie). Andrey tossed a sample implementation into
newsham's 9p library. We  saw a 27x improvement in performance from
calgary to sandia for a big file. Fcp did not come close.

There's work to be done; 9p is great, but it's not the end of protocols.

ron

Re: [9fans] A simple experiment

[erik quanstrom]

> And, in the end, if what you want is a stream, then discrete
> request/response transactions can be made to work, but they don't
> always work as well.

streams are an illusion of the transport layer.

- erik

Re: [9fans] A simple experiment

[Russ Cox]

On Wed, Apr 28, 2010 at 12:06 PM, erik quanstrom
<quanstro@labs.coraid.com> wrote:
>> And, in the end, if what you want is a stream, then discrete
>> request/response transactions can be made to work, but they don't
>> always work as well.
>
> streams are an illusion of the transport layer.

packets, bytes, and bits are illusions too.

russ

Re: [9fans] A simple experiment

[Francisco J Ballesteros]

> We did a simple experiment recently: added a new 9p type called
> Tstream, because this issue of streams vs. transactions has been
> bugging me for years. The semantics are simple: it's a lot like Tread
> (almost same packet) but a single Tstream results in any number of
> Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
> (/usr/rminnich/movie). Andrey tossed a sample implementation into
> newsham's 9p library. We  saw a 27x improvement in performance from
> calgary to sandia for a big file. Fcp did not come close.

That's similar to a Tget in op with unlimited replies. The difference adds on
quickly.

Re: [9fans] A simple experiment

[ron minnich]

On Wed, Apr 28, 2010 at 1:36 PM, Francisco J Ballesteros <nemo@lsub.org> wrote:

> That's similar to a Tget in op with unlimited replies. The difference adds on
> quickly.

neat, I need to study op more than I did :-)

ron

Re: [9fans] A simple experiment

[EBo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 885 bytes --]

Francisco J Ballesteros <nemo@lsub.org> said:

> > We did a simple experiment recently: added a new 9p type called
> > Tstream, because this issue of streams vs. transactions has been
> > bugging me for years. The semantics are simple: it's a lot like Tread
> > (almost same packet) but a single Tstream results in any number of
> > Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
> > (/usr/rminnich/movie). Andrey tossed a sample implementation into
> > newsham's 9p library. We  saw a 27x improvement in performance from
> > calgary to sandia for a big file. Fcp did not come close.
>
> That's similar to a Tget in op with unlimited replies. The difference adds on
> quickly.

cool.  Is Tstream integrated back into 9p and publicly available?

Also, where can i find out more info about op and Tget?


  Thanks and best regards,

  EBo --

Re: [9fans] A simple experiment

[ron minnich]

On Wed, Apr 28, 2010 at 2:05 PM, EBo <ebo@sandien.com> wrote:

> cool.  Is Tstream integrated back into 9p and publicly available?

no, it's an experiment at this point. there are other ideas out there,
and I should have paid more attention to op as nemo points out :-)

the main point was to see what was possible, given the well known poor
performance of 9p on high latency links. This poor performance has an
analog in HPC, where microseconds are large units of time.

ron

Re: [9fans] A simple experiment

[Gorka Guardiola]

On Wed, Apr 28, 2010 at 11:05 PM, EBo <ebo@sandien.com> wrote:
> Also, where can i find out more info about op and Tget?
>
http://lsub.org/ls/export/opiwp9.pdf
http://lsub.org/ls/export/opiwp9tlk.pdf

--
- curiosity sKilled the cat

Re: [9fans] A simple experiment

[Skip Tavakkolian]

>> We did a simple experiment recently: added a new 9p type called
>> Tstream, because this issue of streams vs. transactions has been
>> bugging me for years. The semantics are simple: it's a lot like Tread
>> (almost same packet) but a single Tstream results in any number of
>> Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
>> (/usr/rminnich/movie). Andrey tossed a sample implementation into
>> newsham's 9p library. We  saw a 27x improvement in performance from
>> calgary to sandia for a big file. Fcp did not come close.
>
> That's similar to a Tget in op with unlimited replies. The difference adds on
> quickly.

i was going to ask if anyone has tried this with op.  is there an
implementation of oxport/ofs for plan 9?

Re: [9fans] A simple experiment

[EBo]

> no, it's an experiment at this point. there are other ideas out there,
> and I should have paid more attention to op as nemo points out :-)

understand.  Don't you just HATE it when you come up with something cool and
then find a different alternative stashed in the literature ;-)  I look
forward to learning more about both.

> the main point was to see what was possible, given the well known poor
> performance of 9p on high latency links. This poor performance has an
> analog in HPC, where microseconds are large units of time.

That is why I got so interested.

  EBo --

Re: [9fans] A simple experiment

[Francisco J Ballesteros]

yes and no.
Yes in that you can use inferno + op to bridge plan 9 systems.
No in that there's no native implementation yet.


On Wed, Apr 28, 2010 at 11:18 PM, Skip Tavakkolian <9nut@9netics.com> wrote:
>>> We did a simple experiment recently: added a new 9p type called
>>> Tstream, because this issue of streams vs. transactions has been
>>> bugging me for years. The semantics are simple: it's a lot like Tread
>>> (almost same packet) but a single Tstream results in any number of
>>> Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
>>> (/usr/rminnich/movie). Andrey tossed a sample implementation into
>>> newsham's 9p library. We  saw a 27x improvement in performance from
>>> calgary to sandia for a big file. Fcp did not come close.
>>
>> That's similar to a Tget in op with unlimited replies. The difference adds on
>> quickly.
>
> i was going to ask if anyone has tried this with op.  is there an
> implementation of oxport/ofs for plan 9?
>
>
>

[9fans] references/citations [was: A simple experiment]

[EBo]

> http://lsub.org/ls/export/opiwp9.pdf
> http://lsub.org/ls/export/opiwp9tlk.pdf

Thanks.  I want to put these into my BibTeX database so that I can cite them
in my thesis.  Do you have BibTeX entries or information regarding where they
were presented or published?

  Thanks and best regards,

  EBo --

Re: [9fans] A simple experiment

[roger peppe]

On 28 April 2010 19:42, ron minnich <rminnich@gmail.com> wrote:
> We did a simple experiment recently: added a new 9p type called
> Tstream, because this issue of streams vs. transactions has been
> bugging me for years. The semantics are simple: it's a lot like Tread
> (almost same packet) but a single Tstream results in any number of
> Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
> (/usr/rminnich/movie). Andrey tossed a sample implementation into
> newsham's 9p library. We  saw a 27x improvement in performance from
> calgary to sandia for a big file. Fcp did not come close.

what happens if the consumer is slow and the Rstream writer
blocks? how do you stop all the other replies on the connection
waiting for the consumer to get on with it?

in fact, how do you stop it deadlocking if the consumer is in
fact waiting for one of those replies?

i suppose this comes down to what the API looks like.
isochronous might be easier, as a slow reader is a bad reader
so you could just throw away some data.

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1954 bytes --]

On Thu, Apr 29, 2010 at 5:40 AM, roger peppe <rogpeppe@gmail.com> wrote:

> On 28 April 2010 19:42, ron minnich <rminnich@gmail.com> wrote:
> > We did a simple experiment recently: added a new 9p type called
> > Tstream, because this issue of streams vs. transactions has been
> > bugging me for years. The semantics are simple: it's a lot like Tread
> > (almost same packet) but a single Tstream results in any number of
> > Rstreams, until you hit no more data (/dev/mouse) or maybe EOF
> > (/usr/rminnich/movie). Andrey tossed a sample implementation into
> > newsham's 9p library. We  saw a 27x improvement in performance from
> > calgary to sandia for a big file. Fcp did not come close.
>
> what happens if the consumer is slow and the Rstream writer
> blocks? how do you stop all the other replies on the connection
> waiting for the consumer to get on with it?
>
> in fact, how do you stop it deadlocking if the consumer is in
> fact waiting for one of those replies?
>
> i suppose this comes down to what the API looks like.
> isochronous might be easier, as a slow reader is a bad reader
> so you could just throw away some data.
>
>
It sounds ok on the surface so far.  Does RStream signal the end of the
stream chunks, or does the TStreamer already know that answer?  Is there any
sort of realtime constraint for handling incoming RStream chunks?   I would
think this could be ok, even if it forces the client to put the streamed
blocks somewhere handy while processing is going on.  Streaming audio and
video over plan 9 links this way might be nice.

But then I start to wonder why we feel we want to compete with HTTP when it
already works, and is still fairly simple.  Nothing wrong with improving 9P
I suppose, but what's so wrong with HTTP transfers that it warrants changing
our beloved resource sharing protocol?  Maybe I'm being too practical, and
not feeling adventurous or something :-)

Dave

[-- Attachment #2: Type: text/html, Size: 2376 bytes --]

Re: [9fans] A simple experiment

[Eric Van Hensbergen]

On Wed, Apr 28, 2010 at 3:51 PM, ron minnich <rminnich@gmail.com> wrote:
> On Wed, Apr 28, 2010 at 1:36 PM, Francisco J Ballesteros <nemo@lsub.org> wrote:
>
>> That's similar to a Tget in op with unlimited replies. The difference adds on
>> quickly.
>
> neat, I need to study op more than I did :-)
>

For the record, I kept telling you that.

    -eric

Re: [9fans] A simple experiment

[erik quanstrom]

> what happens if the consumer is slow and the Rstream writer
> blocks? how do you stop all the other replies on the connection
> waiting for the consumer to get on with it?

the tcp window closes. and the producer blocks.

> in fact, how do you stop it deadlocking if the consumer is in
> fact waiting for one of those replies?

i don't think i understand this.

i think Tstream would be very dependant on the
transport layer.  that's the part that makes me nervous.
all the world's a tcp?

- erik

Re: [9fans] A simple experiment

[erik quanstrom]

> But then I start to wonder why we feel we want to compete with HTTP when it
> already works, and is still fairly simple.  Nothing wrong with improving 9P
> I suppose, but what's so wrong with HTTP transfers that it warrants changing
> our beloved resource sharing protocol?  Maybe I'm being too practical, and
> not feeling adventurous or something :-)

do we put a http bag on the side of every /n/remoteslowlink
fileserver, since 9p can't take care of it's own business?

- erik

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 835 bytes --]

On Thu, Apr 29, 2010 at 7:22 AM, erik quanstrom <quanstro@quanstro.net>wrote:

> > what happens if the consumer is slow and the Rstream writer
> > blocks? how do you stop all the other replies on the connection
> > waiting for the consumer to get on with it?
>
> the tcp window closes. and the producer blocks.
>
> > in fact, how do you stop it deadlocking if the consumer is in
> > fact waiting for one of those replies?
>
> i don't think i understand this.
>
> i think Tstream would be very dependant on the
> transport layer.  that's the part that makes me nervous.
> all the world's a tcp?
>
>
What does 9P require to function?  If TStream has the same or lesser
requirements, then there's no problem right?  This comes back to my
wondering why we don't just use 9P to set up HTTP streams.


> - erik
>
>

[-- Attachment #2: Type: text/html, Size: 1390 bytes --]

Re: [9fans] A simple experiment

[erik quanstrom]

> What does 9P require to function?  If TStream has the same or lesser
> requirements, then there's no problem right?  This comes back to my
> wondering why we don't just use 9P to set up HTTP streams.

see /sys/src/doc/il/il.ps
- reliable,
- in order.
(the other three are not hard requirements)

i would think that Tstream has greater requirements.
it would seem to require flow control.

- erik

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

On Thu, Apr 29, 2010 at 7:43 AM, erik quanstrom <quanstro@quanstro.net>wrote:

> > What does 9P require to function?  If TStream has the same or lesser
> > requirements, then there's no problem right?  This comes back to my
> > wondering why we don't just use 9P to set up HTTP streams.
>
> see /sys/src/doc/il/il.ps
> - reliable,
> - in order.
> (the other three are not hard requirements)
>
> i would think that Tstream has greater requirements.
> it would seem to require flow control.
>
> - erik
>
>

9P doesn't require any flow control?  That doesn't seem right :-)  But then
again it doesn't stream, at least in the traditional way I think of
streaming.  To stream you typically need flow control, so 9P isn't good for
streaming in the sense I think of streaming. (yet?)

Fix 9P or don't is the decision to be made.

[-- Attachment #2: Type: text/html, Size: 1330 bytes --]

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1184 bytes --]

On Thu, Apr 29, 2010 at 7:35 AM, erik quanstrom <quanstro@quanstro.net>wrote:

> > But then I start to wonder why we feel we want to compete with HTTP when
> it
> > already works, and is still fairly simple.  Nothing wrong with improving
> 9P
> > I suppose, but what's so wrong with HTTP transfers that it warrants
> changing
> > our beloved resource sharing protocol?  Maybe I'm being too practical,
> and
> > not feeling adventurous or something :-)
>
> do we put a http bag on the side of every /n/remoteslowlink
> fileserver, since 9p can't take care of it's own business?
>
> - erik
>
> It's not quite fair to say 9P *can't* take care of it.  The question is
"does it take care of it well enough?".

I guess my point is we can change 9P to suit every purpose under the sun if
we want, but then does it still have all the nice simple, properties of 9P?
 Wasn't IL somewhat abandoned because to make it as good as TCP you
basically had to implement TCP anyway?

I'm totally undecided on this, but just asking the question to make sure
everyone feels good about adding streaming to 9P, which seems to indicate a
need to add flow control as well.

Dave

[-- Attachment #2: Type: text/html, Size: 1597 bytes --]

Re: [9fans] A simple experiment

[Gabriel Díaz]

Hello


how well it works with firewalls, address translation, "deep inspection", etc.? never tried il outside home. . .

gabi



----- Original Message ----
From: erik quanstrom <quanstro@quanstro.net>
To: 9fans@9fans.net
Sent: Thu, April 29, 2010 4:43:48 PM
Subject: Re: [9fans] A simple experiment

> What does 9P require to function?  If TStream has the same or lesser
> requirements, then there's no problem right?  This comes back to my
> wondering why we don't just use 9P to set up HTTP streams.

see /sys/src/doc/il/il.ps
- reliable,
- in order.
(the other three are not hard requirements)

i would think that Tstream has greater requirements.
it would seem to require flow control.

- erik

Re: [9fans] A simple experiment

[ron minnich]

On Thu, Apr 29, 2010 at 8:03 AM, David Leimbach <leimy2k@gmail.com> wrote:

> 9P doesn't require any flow control?  That doesn't seem right :-)  But then
> again it doesn't stream, at least in the traditional way I think of
> streaming.  To stream you typically need flow control, so 9P isn't good for
> streaming in the sense I think of streaming. (yet?)
> Fix 9P or don't is the decision to be made.

it's going to get fixed.

ron

Re: [9fans] A simple experiment

[ron minnich]

On Thu, Apr 29, 2010 at 5:40 AM, roger peppe <rogpeppe@gmail.com> wrote:

> what happens if the consumer is slow and the Rstream writer
> blocks? how do you stop all the other replies on the connection
> waiting for the consumer to get on with it?

there are not replies -- the rstream is a reply. If the consumer is
slow packets are lost. That's life.


>
> in fact, how do you stop it deadlocking if the consumer is in
> fact waiting for one of those replies?

??

One Tstream. Lots of Rstreams. Zero interaction.

> i suppose this comes down to what the API looks like.
> isochronous might be easier, as a slow reader is a bad reader
> so you could just throw away some data.

Yes.

ron

Re: [9fans] A simple experiment

[ron minnich]

On Thu, Apr 29, 2010 at 8:06 AM, David Leimbach <leimy2k@gmail.com> wrote:

> I'm totally undecided on this, but just asking the question to make sure
> everyone feels good about adding streaming to 9P, which seems to indicate a
> need to add flow control as well.

the problem of multiplexing several streams onto one stream, with
priority and avoiding head of line blocking, was solved a long time
ago.

Tstream is unlikely to be added to 9p.

ron

Re: [9fans] A simple experiment

[Eric Van Hensbergen]

On Thu, Apr 29, 2010 at 10:16 AM, ron minnich <rminnich@gmail.com> wrote:
> On Thu, Apr 29, 2010 at 8:03 AM, David Leimbach <leimy2k@gmail.com> wrote:
>
>> 9P doesn't require any flow control?  That doesn't seem right :-)  But then
>> again it doesn't stream, at least in the traditional way I think of
>> streaming.  To stream you typically need flow control, so 9P isn't good for
>> streaming in the sense I think of streaming. (yet?)
>> Fix 9P or don't is the decision to be made.
>
> it's going to get fixed.
>

Well, its being fixed.  Whether or not it gets back-ported to Plan 9
is the question :)

      -eric

Re: [9fans] A simple experiment

[erik quanstrom]

>  Wasn't IL somewhat abandoned because to make it as good as TCP you
> basically had to implement TCP anyway?

due to a failure of vision, the internet only does
well with certain types of ip packets.

il is still an excellent protocol for local networks.

> 9P doesn't require any flow control?  That doesn't seem right :-)  But then

9p, like aoe, is a ping-pong protocol.  each message requires an ack.
therefore, the transport layer doesn't need flow control.

- erik

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 702 bytes --]

On Thu, Apr 29, 2010 at 8:34 AM, erik quanstrom <quanstro@quanstro.net>wrote:

> >  Wasn't IL somewhat abandoned because to make it as good as TCP you
> > basically had to implement TCP anyway?
>
> due to a failure of vision, the internet only does
> well with certain types of ip packets.
>
> il is still an excellent protocol for local networks.
>
> > 9P doesn't require any flow control?  That doesn't seem right :-)  But
> then
>
> 9p, like aoe, is a ping-pong protocol.  each message requires an ack.
> therefore, the transport layer doesn't need flow control.
>
> Right, but when you have one T message for many R messages, doesn't that
change things a bit?


> - erik
>
>

[-- Attachment #2: Type: text/html, Size: 1231 bytes --]

Re: [9fans] A simple experiment

[erik quanstrom]

> > 9p, like aoe, is a ping-pong protocol.  each message requires an ack.
> > therefore, the transport layer doesn't need flow control.
> >
> Right, but when you have one T message for many R messages, doesn't that
> change things a bit?

yes.  if you were to add that, then a flow-controlled transport
would be required.

- erik

Re: [9fans] A simple experiment

[Bakul Shah]

On Thu, 29 Apr 2010 11:34:44 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> >  Wasn't IL somewhat abandoned because to make it as good as TCP you
> > basically had to implement TCP anyway?
>
> due to a failure of vision, the internet only does
> well with certain types of ip packets.

:-)

> il is still an excellent protocol for local networks.
>
> > 9P doesn't require any flow control?  That doesn't seem right :-)  But then
>
> 9p, like aoe, is a ping-pong protocol.  each message requires an ack.
> therefore, the transport layer doesn't need flow control.

Therefore, it is also not able to utilise bandwidth
effectively over longhaul links. As an example, US coast
to coast round trip time latency is about 100ms. Now consider
fcp. Each worker thread of fcp does 8K read/writes. Due to
pingponging, the *most* a thread can xfer coast to coast is
80KBps (for 16 threads, 1.28MBps).  It is actually much worse
since each thread doesn't even overlap reads with writes.

Short of a sliding window that is as large as the capacity of
the pipe, you can't expect to keep it full.  As usual one has
to trade off simplicity vs performance.

I do hope 9p evolves.

Re: [9fans] A simple experiment

[ron minnich]

On Thu, Apr 29, 2010 at 10:08 AM, Bakul Shah <bakul+plan9@bitblocks.com> wrote:

> Short of a sliding window that is as large as the capacity of
> the pipe, you can't expect to keep it full.  As usual one has
> to trade off simplicity vs performance.
>
> I do hope 9p evolves.
>
>

The problem revealed by fcp is worse than that. You're rewriting a
simple program (cp) to cover for a lack of capability in the kernel.
There's no reason that I can't do a read(1048576) and have the kernel
turn that into as many reads as possible -- in this case, for 8k, 2^7
reads.

Oh, and, it is likely that when we issue that many or more reads, we
want a flow controlled network. Just guessing :-)

ron

Re: [9fans] A simple experiment

[erik quanstrom]

> > 9p, like aoe, is a ping-pong protocol.  each message requires an ack.
> > therefore, the transport layer doesn't need flow control.
>
> Therefore, it is also not able to utilise bandwidth
> effectively over longhaul links. As an example, US coast
> to coast round trip time latency is about 100ms. Now consider
> fcp. Each worker thread of fcp does 8K read/writes. Due to
> pingponging, the *most* a thread can xfer coast to coast is
> 80KBps (for 16 threads, 1.28MBps).  It is actually much worse
> since each thread doesn't even overlap reads with writes.

i think you are conflating a ping-pong protocol with the
limitation of a single outstanding message.  there is no
reason that one needs a 1:1 correspondence between threads
and outstanding messages.  nor does the protocol inherently
prohibit 1 write from becoming multiple messages.

in the case of aoe, you can have up to 2^32 - 1 outstanding
packets at the same time to each target.  thus the theoretical
maximum would be
	80 * (2^31 - 1) kbps per target.
the aoe driver (in contrib) has a limit of 128 8k jumbo frames
outstanding per target.  the aoe driver has a fixed number of
thread per target.

> Short of a sliding window that is as large as the capacity of
> the pipe, you can't expect to keep it full.  As usual one has
> to trade off simplicity vs performance.

i don't think so.  the only thing a sliding window gives you
is < 1 ack per message sent.

-erik

Re: [9fans] A simple experiment

[erik quanstrom]

> Oh, and, it is likely that when we issue that many or more reads, we
> want a flow controlled network. Just guessing :-)

the downside of course is that the set of suitable protocols
is then greatly reduced.  tftp, rudp, il, cec to name a few simple
protocols (yes i've done 9p over cec) would then be unusable
for 9p.  and the minimum requirements for participating in
a 9p converstation would be increased.  maybe that's okay,
but it's hard to give that up.

one thing that 9p doesn't address (that aoe does) is the maximum
number of outstanding messages.  that can be a fairly effective
means of flow control.

- erik

Re: [9fans] A simple experiment

[Tim Newsham]

> But then I start to wonder why we feel we want to compete with HTTP when it
> already works, and is still fairly simple.  Nothing wrong with improving 9P
> I suppose, but what's so wrong with HTTP transfers that it warrants changing
> our beloved resource sharing protocol?  Maybe I'm being too practical, and
> not feeling adventurous or something :-)

See the "op" papers for their justification.

> Dave

Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Thu, Apr 29, 2010 at 10:48 AM, Tim Newsham <newsham@lava.net> wrote:

> But then I start to wonder why we feel we want to compete with HTTP when it
>> already works, and is still fairly simple.  Nothing wrong with improving
>> 9P
>> I suppose, but what's so wrong with HTTP transfers that it warrants
>> changing
>> our beloved resource sharing protocol?  Maybe I'm being too practical, and
>> not feeling adventurous or something :-)
>>
>
> See the "op" papers for their justification.
>

Will do!


>
>  Dave
>>
>
> Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com
>
>

[-- Attachment #2: Type: text/html, Size: 1392 bytes --]

Re: [9fans] A simple experiment

[EBo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 605 bytes --]


> >> 9P doesn't require any flow control? �That doesn't seem right :-) �But then
> >> again it doesn't stream, at least in the traditional way I think of
> >> streaming. �To stream you typically need flow control, so 9P isn't good for
> >> streaming in the sense I think of streaming. (yet?)
> >> Fix 9P or don't is the decision to be made.
> >
> > it's going to get fixed.
>
> Well, its being fixed.  Whether or not it gets back-ported to Plan 9
> is the question :)

Where is the fix happening?  Maybe there are people around that are willing to
help back-port  :-)

  EBo --

Re: [9fans] A simple experiment

[Skip Tavakkolian]

i think in almost all cases putting Op between 9P endpoints will
solve the slowness problem of high rtt networks.

>> But then I start to wonder why we feel we want to compete with HTTP when it
>>> already works, and is still fairly simple.  Nothing wrong with improving
>>> 9P
>>> I suppose, but what's so wrong with HTTP transfers that it warrants
>>> changing
>>> our beloved resource sharing protocol?  Maybe I'm being too practical, and
>>> not feeling adventurous or something :-)
>>>
>>
>> See the "op" papers for their justification.
>>
>
> Will do!
>
>
>>
>>  Dave
>>>
>>
>> Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com
>>
>>

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

Hmmm is Op the same Op from Octopus?

Just making sure.

Dave

On Thu, Apr 29, 2010 at 11:41 AM, Skip Tavakkolian <9nut@9netics.com> wrote:

> i think in almost all cases putting Op between 9P endpoints will
> solve the slowness problem of high rtt networks.
>
> >> But then I start to wonder why we feel we want to compete with HTTP when
> it
> >>> already works, and is still fairly simple.  Nothing wrong with
> improving
> >>> 9P
> >>> I suppose, but what's so wrong with HTTP transfers that it warrants
> >>> changing
> >>> our beloved resource sharing protocol?  Maybe I'm being too practical,
> and
> >>> not feeling adventurous or something :-)
> >>>
> >>
> >> See the "op" papers for their justification.
> >>
> >
> > Will do!
> >
> >
> >>
> >>  Dave
> >>>
> >>
> >> Tim Newsham | www.thenewsh.com/~newsham | thenewsh.blogspot.com
> >>
> >>
>
>
>

[-- Attachment #2: Type: text/html, Size: 1542 bytes --]

Re: [9fans] A simple experiment

[Lyndon Nerenberg]

> due to a failure of vision, the internet only does
> well with certain types of ip packets.

Well now *there* is a sweeping statement about the state of the universe
circa 1980. Care to elaborate a teensy bit?

Re: [9fans] A simple experiment

[erik quanstrom]

On Thu Apr 29 14:53:56 EDT 2010, lyndon@orthanc.ca wrote:
> > due to a failure of vision, the internet only does
> > well with certain types of ip packets.
>
> Well now *there* is a sweeping statement about the state of the universe
> circa 1980. Care to elaborate a teensy bit?

if you pick a random protocol type that's not tcp or udp and
the chances your little packets will make it across the intertubes
will be lowered quite a bit.  this isn't the ip designer's fault.

although for giggles i just set this up:

	minooka; aux/listen1 -tv /net.alt/il!*!4000 /386/bin/cat
	listen started
	incoming call for /net.alt/il!*!4000 from 74.166.29.253 in /net.alt/il/1

and surprise, surprise, i can call out through a bridge and a
telco router and touch it

	; telnet il!minooka.coraid.com!4000
	connected to il!minooka.coraid.com!4000 on /net/il/3
	hello, world
	hello, world<cr>

i'll leave that up for a while.  have fun.  i'll be interested what
the success rate will be.

- erik

Re: [9fans] A simple experiment

[C H Forsyth]

>But then I start to wonder why we feel we want to compete with HTTP when it
>already works, and is still fairly simple.

http is by no means simple, although yes, it could be still more complicated.

Re: [9fans] A simple experiment

[Skip Tavakkolian]

>> due to a failure of vision, the internet only does
>> well with certain types of ip packets.
>
> Well now *there* is a sweeping statement about the state of the universe
> circa 1980. Care to elaborate a teensy bit?

i think the point is IL v. TCP;  this is dejavu all over again:

http://groups.google.com/group/comp.os.plan9/browse_thread/thread/23484eb901f3d8e7/d261b5a523bf53d1?lnk=gst&q=why+IL#d261b5a523bf53d1

Re: [9fans] A simple experiment

[Skip Tavakkolian]

> Hmmm is Op the same Op from Octopus?

yes. nemo has put the op specific stuff here:

/n/sources/contrib/nemo/octopus/op.src.tgz

Re: [9fans] A simple experiment

[Skip Tavakkolian]

fyi:
from rangboom.com network (colo at a ISP):

	cpu% telnet il!minooka.coraid.com!4000
	connected to il!minooka.coraid.com!4000 on /net/il/3

from 9netics.com (centurytel DSL): still waiting :)

> On Thu Apr 29 14:53:56 EDT 2010, lyndon@orthanc.ca wrote:
>> > due to a failure of vision, the internet only does
>> > well with certain types of ip packets.
>>
>> Well now *there* is a sweeping statement about the state of the universe
>> circa 1980. Care to elaborate a teensy bit?
>
> if you pick a random protocol type that's not tcp or udp and
> the chances your little packets will make it across the intertubes
> will be lowered quite a bit.  this isn't the ip designer's fault.
>
> although for giggles i just set this up:
>
> 	minooka; aux/listen1 -tv /net.alt/il!*!4000 /386/bin/cat
> 	listen started
> 	incoming call for /net.alt/il!*!4000 from 74.166.29.253 in /net.alt/il/1
>
> and surprise, surprise, i can call out through a bridge and a
> telco router and touch it
>
> 	; telnet il!minooka.coraid.com!4000
> 	connected to il!minooka.coraid.com!4000 on /net/il/3
> 	hello, world
> 	hello, world<cr>
>
> i'll leave that up for a while.  have fun.  i'll be interested what
> the success rate will be.
>
> - erik

Re: [9fans] A simple experiment

[erik quanstrom]

On Thu Apr 29 15:55:44 EDT 2010, 9nut@9netics.com wrote:
> fyi:
> from rangboom.com network (colo at a ISP):
>
> 	cpu% telnet il!minooka.coraid.com!4000
> 	connected to il!minooka.coraid.com!4000 on /net/il/3
>
> from 9netics.com (centurytel DSL): still waiting :)

interesting.  i was able to get there on both my
residential dsl lines.  (att & negia.net)

thanks for the il thread.  does anyone know what
happened to the nat code that eric grosse mentioned?

- erik

Re: [9fans] A simple experiment

[Skip Tavakkolian]

>>But then I start to wonder why we feel we want to compete with HTTP when it
>>already works, and is still fairly simple.
>
> http is by no means simple, although yes, it could be still more complicated.

they'll stop when the spec is massive enough to undergo gravitation
collapse.

Re: [9fans] A simple experiment

[erik quanstrom]

> On Thu Apr 29 15:55:44 EDT 2010, 9nut@9netics.com wrote:
> > fyi:
> > from rangboom.com network (colo at a ISP):
> >
> > 	cpu% telnet il!minooka.coraid.com!4000
> > 	connected to il!minooka.coraid.com!4000 on /net/il/3
> >
> > from 9netics.com (centurytel DSL): still waiting :)
>
> interesting.  i was able to get there on both my
> residential dsl lines.  (att & negia.net)
>
> thanks for the il thread.  does anyone know what
> happened to the nat code that eric grosse mentioned?

rather than fiddle with web server configuration, telneting
to that port will now return a list of all connections.

- erik

Re: [9fans] A simple experiment

[Christopher Nielsen]

On Thu, Apr 29, 2010 at 08:06, Gabriel Díaz <gdiaz@rejaa.com> wrote:
> Hello
>
>
> how well it works with firewalls, address translation, "deep inspection", etc.? never tried il outside home. . .

It doesn't play well with firewalls, NAT, or deep inspection because
none of the vendors have added support for it. I tried to get Cisco to
add IL support back in 2001, but they politely refused.

-- 
Christopher Nielsen
"They who can give up essential liberty for temporary safety, deserve
neither liberty nor safety." --Benjamin Franklin
"The tree of liberty must be refreshed from time to time with the
blood of patriots & tyrants." --Thomas Jefferson

Re: [9fans] A simple experiment

[erik quanstrom]

> It doesn't play well with firewalls, NAT, or deep inspection because
> none of the vendors have added support for it. I tried to get Cisco to
> add IL support back in 2001, but they politely refused.

oddly, i'm not having trouble going through 5 different nats
over 2 residential and 2 business dsl lines.  one of the connections
traverses 2 nats before reaching the intertubes.

- erik

Re: [9fans] A simple experiment

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

On Thu, Apr 29, 2010 at 12:44 PM, C H Forsyth <forsyth@vitanuova.com> wrote:

> >But then I start to wonder why we feel we want to compete with HTTP when
> it
> >already works, and is still fairly simple.
>
> http is by no means simple, although yes, it could be still more
> complicated.
>
>
I guess I think it's simple because of the crap I work with all day...
everything's relative?

[-- Attachment #2: Type: text/html, Size: 681 bytes --]

Re: [9fans] A simple experiment

[hiro]

There are a few dsl's which require you to use their routers with NAT
and other stuff, but I've never tried these out, I've always had real
IP connections here...

As long as we have enough ipv4 adresses I would say: Build your own
firewall, router or NAT...

On 4/29/10, Christopher Nielsen <cnielsen@pobox.com> wrote:
> On Thu, Apr 29, 2010 at 08:06, Gabriel Díaz <gdiaz@rejaa.com> wrote:
>> Hello
>>
>>
>> how well it works with firewalls, address translation, "deep inspection",
>> etc.? never tried il outside home. . .
>
> It doesn't play well with firewalls, NAT, or deep inspection because
> none of the vendors have added support for it. I tried to get Cisco to
> add IL support back in 2001, but they politely refused.
>
> --
> Christopher Nielsen
> "They who can give up essential liberty for temporary safety, deserve
> neither liberty nor safety." --Benjamin Franklin
> "The tree of liberty must be refreshed from time to time with the
> blood of patriots & tyrants." --Thomas Jefferson
>
>

Re: [9fans] A simple experiment

[Christopher Nielsen]

On Thu, Apr 29, 2010 at 13:40, erik quanstrom <quanstro@quanstro.net> wrote:
>> It doesn't play well with firewalls, NAT, or deep inspection because
>> none of the vendors have added support for it. I tried to get Cisco to
>> add IL support back in 2001, but they politely refused.
>
> oddly, i'm not having trouble going through 5 different nats
> over 2 residential and 2 business dsl lines.  one of the connections
> traverses 2 nats before reaching the intertubes.

Interesting. I've seen quite a few problems in the past that were all
traced back to NAT or a firewall not handling IL properly. Admittedly,
I haven't used IL recently, so maybe something has changed. It's good
to hear that you're not having trouble.

-- 
Christopher Nielsen
"They who can give up essential liberty for temporary safety, deserve
neither liberty nor safety." --Benjamin Franklin
"The tree of liberty must be refreshed from time to time with the
blood of patriots & tyrants." --Thomas Jefferson

Re: [9fans] A simple experiment

[Derek Fawcus]

On Thu, Apr 29, 2010 at 01:32:23PM -0700, Christopher Nielsen wrote:
> It doesn't play well with firewalls, NAT, or deep inspection because
> none of the vendors have added support for it. I tried to get Cisco to
> add IL support back in 2001, but they politely refused.

Add support to what?  Also what level of 'support'?

IOS should already support IL in access lists simply by virtue of the
fact that one can specify a numeric IP protocol.

I agree that NAT and stateful firewalls (e.g. 'ip inspect' in IOS)
would need explicit support to understand the packet layout.

But one can always add exceptions to the firewall rules to allow
IL through uninspected.  Thats what I do on my IOS routers for
oddball protocols.  NAT - it should simply die,  until then
run IL over IPv6 and avoid NAT?

Re: [9fans] A simple experiment

[erik quanstrom]

> I agree that NAT and stateful firewalls (e.g. 'ip inspect' in IOS)
> would need explicit support to understand the packet layout.

what il services would you apply spi to?  one doesn't
ftp or http over il.

> NAT - it should simply die,  until then
> run IL over IPv6 and avoid NAT?

il isn't defined over ip6.

why should nat die?  translating network addresses
from one network to another seems natural enough
to me—and quite similar to what various storage systems
do to present logical volumes.  why should renumber a formerly
private network because i'd like to hook it up to
the internet?  why should i renumber my network
because i change service providers?  why is using nat
to make many hosts look like one a bad thing?

- erik

Re: [9fans] A simple experiment

[Anthony Sorace]

[-- Attachment #1: Type: text/plain, Size: 1398 bytes --]

ron said:

> Oh, and, it is likely that when we issue that many or more reads, we
> want a flow controlled network. Just guessing :-)


You'd want one, sure. But (unless I'm missing something) it doesn't
seem that increasing the number of outstanding messages would
*require* it (your performance would just suck otherwise). If you
treat the number of outstanding requests as a tunable (per mount, per
network type) you'd be able to use the same protocol over minimal
links as on tcp.

dave said:

> Wasn't IL somewhat abandoned because to make it as good
> as TCP you basically had to implement TCP anyway?

My impression is that it was pulled from mainline mostly because it
was seen as being too much work to maintain for a small group. It's
also not really sensible to talk about "as good as" without defining
your context. Over fast, local networks, IL is faster than TCP. It's
not as generic, but that's not the same as "worse".

erik said:

> why is using nat to make many hosts look like one a bad thing?

I suspect the reaction is based on being forced to use it when you'd
rather not, like many residential ISPs require. It's particularly
upsetting when the CPE doesn't even have a globally routable address.

aside: i made the il connection. i think you've got either good luck
or a good config on your end. it didn't want to ask dns for some reason.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [9fans] A simple experiment

[Bakul Shah]

On Thu, 29 Apr 2010 13:23:00 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> > > 9p, like aoe, is a ping-pong protocol.  each message requires an ack.
> > > therefore, the transport layer doesn't need flow control.
> >
> > Therefore, it is also not able to utilise bandwidth
> > effectively over longhaul links. As an example, US coast
> > to coast round trip time latency is about 100ms. Now consider
> > fcp. Each worker thread of fcp does 8K read/writes. Due to
> > pingponging, the *most* a thread can xfer coast to coast is
> > 80KBps (for 16 threads, 1.28MBps).  It is actually much worse
> > since each thread doesn't even overlap reads with writes.
>
> i think you are conflating a ping-pong protocol with the
> limitation of a single outstanding message.  there is no
> reason that one needs a 1:1 correspondence between threads
> and outstanding messages.  nor does the protocol inherently
> prohibit 1 write from becoming multiple messages.

What I am saying is that RPC (which is how 9p is mostly used)
has this inherent performance limitation due to latency.  To
get around that if you use multiple outstanding messages
(regardless of one per thread or many per thread), you *will*
require flow control or you run into problems -- imagine
having thousands of outstanding messages. Or imagine having a
low bandwidth link for fcp.  Fcp will hog the link, to the
detriment of other programs.  So on the high end fcp
throughput is limited 1.28MBps, on the low end it will hog
the link completey (unless your kernel implements some sort
of fair queuing)!  If you want to make optimum use of the
available bandwidth while adjusting to changing conditions
and not competely clog up the pipe, you need a feedback
mechanism and window opening algorithm ALA TCP or something.

And anyway why do all this in fcp or every other program that
needs streaming?  It should be factored out.

> > Short of a sliding window that is as large as the capacity of
> > the pipe, you can't expect to keep it full.  As usual one has
> > to trade off simplicity vs performance.
>
> i don't think so.  the only thing a sliding window gives you
> is < 1 ack per message sent.

If the sender->receiver pipe can hold N bytes and the sender
is streaming (that is, keeping the pipe full), the sender
*will* be ahead of the receiver by N bytes. So a *streaming*
protcol has to allow it to be N bytes ahead.

Even if you have one ack per message, in a sliding window of
N messages (or bytes), the sender is allowed to get ahead of
the receiver by upto N more messages (or bytes). Here I am
not concerned about in-order delivery (though typically
people assume this when they talk about sliding windows).

If you assume in-order delivery you can coalesce multiple
acks but that can be seen as an ack optimization (but then
either you throw away out of order deliveries or have to add
selective ack or something).

Re: [9fans] A simple experiment

[erik quanstrom]

> My impression is that it was pulled from mainline mostly because it
> was seen as being too much work to maintain for a small group. It's
> also not really sensible to talk about "as good as" without defining
> your context. Over fast, local networks, IL is faster than TCP. It's
> not as generic, but that's not the same as "worse".

history -fD /n/sources/plan9/sys/src/9/ip/il.c reveals just 4 interesting
diffs since 2002.

> > why is using nat to make many hosts look like one a bad thing?
>
> I suspect the reaction is based on being forced to use it when you'd
> rather not, like many residential ISPs require. It's particularly
> upsetting when the CPE doesn't even have a globally routable address.

that's like saying ferraris are bad because they can be driven into trees.
sometimes you really want many hosts to look like one.  for example,
you may want a farm of http servers may hide behind a single ip address.
this is a good thing for scalability.

> aside: i made the il connection. i think you've got either good luck
> or a good config on your end. it didn't want to ask dns for some reason.

cs?  il was removed from the protocols cs knows about.  try
cs from /n/sourcesdump/2006/0101/plan9/386/bin/cs.

if that fixes it, i'll fix 9atom.

- erik

Re: [9fans] A simple experiment

[erik quanstrom]

> If the sender->receiver pipe can hold N bytes and the sender
> is streaming (that is, keeping the pipe full), the sender
> *will* be ahead of the receiver by N bytes. So a *streaming*
> protcol has to allow it to be N bytes ahead.

i don't think this is the normal definition.

> Even if you have one ack per message, in a sliding window of
> N messages (or bytes), the sender is allowed to get ahead of
> the receiver by upto N more messages (or bytes). Here I am
> not concerned about in-order delivery (though typically
> people assume this when they talk about sliding windows).

a sliding window is just a particular implementation  what's
required is many messages in flight.

aoe for example, has a maximum outstanding (set by the target).
with good client-side congestion control, tracking a window
is not helpful.  the client (sender) knows best.

aoe has no sequence number; just a tag.  the target (receiver)
just echoes tags and is allowed no interpretation.  they work
like fids.  i don't think this qualifies as a sliding window.

yet the tcp congestion algorithm works well with this setup.
multiple gbe links can be saturated and yet congestion is still
handled.

i don't see any reason why 9p couldn't use some of the same
congestion control ideas.  the trick would be to feed back packet loss
detection and retransmission info to the point where file io gets
turned into rpcs—the mount driver

- erik

Re: [9fans] A simple experiment

[Charles Forsyth]

>I agree that NAT and stateful firewalls (e.g. 'ip inspect' in IOS)
>would need explicit support to understand the packet layout.

it's a pity that the protocol revisions didn't move the port-addressing
function outside individual protocols.

Re: [9fans] A simple experiment

[hiro]

> erik said:
>
>> why is using nat to make many hosts look like one a bad thing?
>
> I suspect the reaction is based on being forced to use it when you'd
> rather not, like many residential ISPs require. It's particularly
> upsetting when the CPE doesn't even have a globally routable address.

How much is many?

Re: [9fans] A simple experiment

[Anthony Sorace]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

>> I suspect the reaction is based on being forced to use it when you'd
>> rather not, like many residential ISPs require. It's particularly
>> upsetting when the CPE doesn't even have a globally routable address.
>
> How much is many?

Um, some? I don't have any sort of global count. I've had 7 broadband
ISPs.
4 of the first 5 (I've stopped asking) wouldn't provide multiple IP
addresses
without moving to a "business-class" service, which is usually bundled
with
one or more things I didn't want (or want to pay for), like symmetric
links,
QoS terms, &c. 2 of the 7 (in the first three; maybe this practice has
finally
been discredited?) didn't provide the CPE with globally-routable IPs.
The
sample size is small, but is distributed across two countries and
three US
states. I've also gotten ample confirmation from others about the
first part.

And, of course, this doesn't address things like hotels and wifi
hotspots,
where the mandated use of nat is so common as to just be assumed.


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 201 bytes --]

Re: [9fans] A simple experiment

[Bakul Shah]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1516 bytes --]

On Fri, 30 Apr 2010 01:01:59 EDT erik quanstrom <quanstro@quanstro.net>  wrote:
> > If the sender->receiver pipe can hold N bytes and the sender
> > is streaming (that is, keeping the pipe full), the sender
> > *will* be ahead of the receiver by N bytes. So a *streaming*
> > protcol has to allow it to be N bytes ahead.
>
> i don't think this is the normal definition.
>
> > Even if you have one ack per message, in a sliding window of
> > N messages (or bytes), the sender is allowed to get ahead of
> > the receiver by upto N more messages (or bytes). Here I am
> > not concerned about in-order delivery (though typically
> > people assume this when they talk about sliding windows).
>
> a sliding window is just a particular implementation  what's
> required is many messages in flight.

I was using sliding window as a shorthand for "allowing the
sender to be N bytes or messages ahead without regard to
sequencing" but yes, the definition I used  is not the
generally accepted defn so I will use yours!

> i don't see any reason why 9p couldn't use some of the same
> congestion control ideas.  the trick would be to feed back packet loss
> detection and retransmission info to the point where file io gets
> turned into rpcs—the mount driver

Agreed -- sort of what I meant by "I hope 9p evolves".
Though I'd probably stick this in a layer below 9p. It can
backpressure a 9p client if it tries to send too much (either
by blocking or returning with equiv of EWOULDBLOCK).

Re: [9fans] A simple experiment

[erik quanstrom]

> > i don't see any reason why 9p couldn't use some of the same
> > congestion control ideas.  the trick would be to feed back packet loss
> > detection and retransmission info to the point where file io gets
> > turned into rpcs→the mount driver
>
> Agreed -- sort of what I meant by "I hope 9p evolves".
> Though I'd probably stick this in a layer below 9p.

i suggested putting it above 9p.  i don't understand your
suggestion.  how would a layer below know that there is
more data?

> It can
> backpressure a 9p client if it tries to send too much (either
> by blocking or returning with equiv of EWOULDBLOCK).

isn't that what you just complained about—fcp having to
do the kernel's work?

i think EWOULDBLOCK would be a big mistake.  io
in plan 9 is synchronous.  and this is very much on purpose.

- erik

Re: [9fans] A simple experiment

[Bakul Shah]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2510 bytes --]

On Fri, 30 Apr 2010 12:13:59 EDT erik quanstrom <quanstro@labs.coraid.com>  wrote:
> > > i don't see any reason why 9p couldn't use some of the same
> > > congestion control ideas.  the trick would be to feed back packet loss
> > > detection and retransmission info to the point where file io gets
> > > turned into rpcs→the mount driver
> >
> > Agreed -- sort of what I meant by "I hope 9p evolves".
> > Though I'd probably stick this in a layer below 9p.
>
> i suggested putting it above 9p.  i don't understand your
> suggestion.  how would a layer below know that there is
> more data?

I was just musing aloud -- just some vague ideas.  9p already
allows 2^16 outstanding messages so in theory one can turn a
read() or write() into a sequence of 9p messsages and only
wait for acks at the end (but process any acks as they
arrive).  Perhaps ack receipt time can be used to measure
latency & bandwidth to transparently control how many 9p
messages can be blasted. The other vague thought was that one
can do this:

	char buf[SOME_LARGE_CONSTANT];
	int count = read(src, buf, sizeof buf);
	if (count > 0) write(dst, buf, count);

But read will wait until buf is filled up or file ends &
write will wait until the entire buf is sent or there is an
error. Now may be this is good enough in that a reading thread
can read in big gulps and pass on filled buffers to a writing
thread. And the writing thread writes out and sends empty
buffers back to the reader.  But one can also think of a
lower level facility that in effect "offload" actual transfer
to a coprocessor.  This allows read()/write() to return much
quicker (but obviously some errors will be delayed). One can
implement something like this using mmap(). Now I am well
aware of issues with mmap (we don't have to go through that
discussion again) but it can be an option for improving
throughput.

Neither of these ideas requires changes to 9p per se.  To
take this any further, I would want to prototype something.
I have coded some bits in Scheme but no time really to do
much.

In turn some examples to explain your suggestion would be
helpful.

> > It can
> > backpressure a 9p client if it tries to send too much (either
> > by blocking or returning with equiv of EWOULDBLOCK).
>
> isn't that what you just complained about—fcp having to
> do the kernel's work?

Blocking would backpressure in that fcp wouldn't be able to
send more data! fcp doesn't have to do anything more.

Re: [9fans] A simple experiment

[Akshat Kumar]

Doesn't make it past my Vonage modem/router.
It would otherwise probably make it past the
Cable modem.

On Thu, Apr 29, 2010 at 12:03 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> if you pick a random protocol type that's not tcp or udp and
> the chances your little packets will make it across the intertubes
> will be lowered quite a bit.  this isn't the ip designer's fault.
>
> although for giggles i just set this up:
>
>        minooka; aux/listen1 -tv /net.alt/il!*!4000 /386/bin/cat
>        listen started
>        incoming call for /net.alt/il!*!4000 from 74.166.29.253 in /net.alt/il/1
>
> - erik