Slow cp, support for 9P2000.s

Slow cp, support for 9P2000.s

[Romano]

On my local wireless network at home between my work laptop and my rpi 4, I tested using 'cp' while in drawterm.  I ran drawterm -G on my rpi, and used srvfs to export its file system. I ran drawterm on my work laptop, and mounted the exported filesystem as /n/pi.  I then timed a 20MB file from /mnt/term/root/ on my work laptop to the /n/pi: 0.20u 0.82s 70.14r. 70s is atrociously slow for a local copy when compared to http or scp, so I decided to look to see if this was known and if any work had been done on this problem for 9p.  I came across John Floren's thesis from 2010[1] and his testing showed that increased latency over 9p significantly increased the time files took to transfer. He also shows modifications to 'cp' and 'exportfs' to allow for streams to make 9p have similar speeds as http. 

So considering Floren's research, I then decided to look at my latency on my local network, and sure enough, I'm getting the following ping results (for 144 packets transmitted, with 1 packet lost): round-trip min/avg/max/stddev = 2.273/15.264/497.869/44.018 ms . With an average latency of 15ms, the time it took for  20MB file to transfer over 9p roughly matches Floren's tests.

I think that modifying 'cp' and 'exportfs' would resolve this problem and will look at patching it per Floren's thesis to confirm.  Would modifying 'cp' and 'exportfs' to allow for streams be something others would be interested in? Or has this already been discussed and the stream solution been found wanting for some technical reason? The protocol, as mentioned in the paper, is called 9P2000.s. I don't see that flavor listed here: http://9p.cat-v.org/implementations

[1] https://pdfs.semanticscholar.org/9b47/22e251df61b4abbe0da61720e57ac7b62f61.pdf

Re: [9front] Slow cp, support for 9P2000.s

[aiju]

Yes his solution has been discussed and found wanting. If I remember right 
it breaks the syscall interface and forces 9P to run over TCP.

There have been ongoing talks about our own protocol 9P2020 to solve this 
problem, but little progress because it's a hard problem to solve properly 
without messing up Plan 9 too much.

In the meantime, `fcp` is fast cp.

- aiju



On Fri, 24 Apr 2020, Romano wrote:

> On my local wireless network at home between my work laptop and my rpi 4, I tested using 'cp' while in drawterm.  I ran drawterm -G on my rpi, and used srvfs to export its file system. I ran drawterm on my work laptop, and mounted the exported filesystem as /n/pi.  I then timed a 20MB file from /mnt/term/root/ on my work laptop to the /n/pi: 0.20u 0.82s 70.14r. 70s is atrociously slow for a local copy when compared to http or scp, so I decided to look to see if this was known and if any work had been done on this problem for 9p.  I came across John Floren's thesis from 2010[1] and his testing showed that increased latency over 9p significantly increased the time files took to transfer. He also shows modifications to 'cp' and 'exportfs' to allow for streams to make 9p have similar speeds as http.
>
> So considering Floren's research, I then decided to look at my latency on my local network, and sure enough, I'm getting the following ping results (for 144 packets transmitted, with 1 packet lost): round-trip min/avg/max/stddev = 2.273/15.264/497.869/44.018 ms . With an average latency of 15ms, the time it took for  20MB file to transfer over 9p roughly matches Floren's tests.
>
> I think that modifying 'cp' and 'exportfs' would resolve this problem and will look at patching it per Floren's thesis to confirm.  Would modifying 'cp' and 'exportfs' to allow for streams be something others would be interested in? Or has this already been discussed and the stream solution been found wanting for some technical reason? The protocol, as mentioned in the paper, is called 9P2000.s. I don't see that flavor listed here: http://9p.cat-v.org/implementations
>
> [1] https://pdfs.semanticscholar.org/9b47/22e251df61b4abbe0da61720e57ac7b62f61.pdf
>

Re: [9front] Slow cp, support for 9P2000.s

[Romano]

Thank you, Aiju!

Using fcp , I get 0.36u 0.70s 25.66r, so more than halving the time. And in retrospect I should have read the man page for cp first, where I would have seen fcp.

I lurk on reddit.com, so no account. I see there's reddit.com/r/9P2020, but it's private. Is the discussion re: 9P2020 mostly there, or elsewhere? I grepped 9P2020 under /n/lists/9front and only saw your message to me.



On April 24, 2020 4:28:15 AM UTC, aiju <aiju@phicode.de> wrote:
>Yes his solution has been discussed and found wanting. If I remember
>right 
>it breaks the syscall interface and forces 9P to run over TCP.
>
>There have been ongoing talks about our own protocol 9P2020 to solve
>this 
>problem, but little progress because it's a hard problem to solve
>properly 
>without messing up Plan 9 too much.
>
>In the meantime, `fcp` is fast cp.
>
>- aiju
>
>
>
>On Fri, 24 Apr 2020, Romano wrote:
>
>> On my local wireless network at home between my work laptop and my
>rpi 4, I tested using 'cp' while in drawterm.  I ran drawterm -G on my
>rpi, and used srvfs to export its file system. I ran drawterm on my
>work laptop, and mounted the exported filesystem as /n/pi.  I then
>timed a 20MB file from /mnt/term/root/ on my work laptop to the /n/pi:
>0.20u 0.82s 70.14r. 70s is atrociously slow for a local copy when
>compared to http or scp, so I decided to look to see if this was known
>and if any work had been done on this problem for 9p.  I came across
>John Floren's thesis from 2010[1] and his testing showed that increased
>latency over 9p significantly increased the time files took to
>transfer. He also shows modifications to 'cp' and 'exportfs' to allow
>for streams to make 9p have similar speeds as http.
>>
>> So considering Floren's research, I then decided to look at my
>latency on my local network, and sure enough, I'm getting the following
>ping results (for 144 packets transmitted, with 1 packet lost):
>round-trip min/avg/max/stddev = 2.273/15.264/497.869/44.018 ms . With
>an average latency of 15ms, the time it took for  20MB file to transfer
>over 9p roughly matches Floren's tests.
>>
>> I think that modifying 'cp' and 'exportfs' would resolve this problem
>and will look at patching it per Floren's thesis to confirm.  Would
>modifying 'cp' and 'exportfs' to allow for streams be something others
>would be interested in? Or has this already been discussed and the
>stream solution been found wanting for some technical reason? The
>protocol, as mentioned in the paper, is called 9P2000.s. I don't see
>that flavor listed here: http://9p.cat-v.org/implementations
>>
>> [1]
>https://pdfs.semanticscholar.org/9b47/22e251df61b4abbe0da61720e57ac7b62f61.pdf
>>

Re: [9front] Slow cp, support for 9P2000.s

[aiju]

None of the main team use reddit for Plan 9 stuff, afaik.

9P2020 (fwiw, not the only name that's been used) has been discussed on 
IRC (freenode, #cat-v) extensively in the past.

aiju

On Fri, 24 Apr 2020, Romano wrote:

> Thank you, Aiju!
> 
> Using fcp , I get 0.36u 0.70s 25.66r, so more than halving the time. And in retrospect I should have read the man page for cp first, where I would have seen fcp.
> 
> I lurk on reddit.com, so no account. I see there's reddit.com/r/9P2020, but it's private. Is the discussion re: 9P2020 mostly there, or elsewhere? I grepped 9P2020 under /n/lists/9front and only saw your message to me.
> 
> 
> 
> On April 24, 2020 4:28:15 AM UTC, aiju <aiju@phicode.de> wrote:
> >Yes his solution has been discussed and found wanting. If I remember
> >right 
> >it breaks the syscall interface and forces 9P to run over TCP.
> >
> >There have been ongoing talks about our own protocol 9P2020 to solve
> >this 
> >problem, but little progress because it's a hard problem to solve
> >properly 
> >without messing up Plan 9 too much.
> >
> >In the meantime, `fcp` is fast cp.
> >
> >- aiju
> >
> >
> >
> >On Fri, 24 Apr 2020, Romano wrote:
> >
> >> On my local wireless network at home between my work laptop and my
> >rpi 4, I tested using 'cp' while in drawterm.  I ran drawterm -G on my
> >rpi, and used srvfs to export its file system. I ran drawterm on my
> >work laptop, and mounted the exported filesystem as /n/pi.  I then
> >timed a 20MB file from /mnt/term/root/ on my work laptop to the /n/pi:
> >0.20u 0.82s 70.14r. 70s is atrociously slow for a local copy when
> >compared to http or scp, so I decided to look to see if this was known
> >and if any work had been done on this problem for 9p.  I came across
> >John Floren's thesis from 2010[1] and his testing showed that increased
> >latency over 9p significantly increased the time files took to
> >transfer. He also shows modifications to 'cp' and 'exportfs' to allow
> >for streams to make 9p have similar speeds as http.
> >>
> >> So considering Floren's research, I then decided to look at my
> >latency on my local network, and sure enough, I'm getting the following
> >ping results (for 144 packets transmitted, with 1 packet lost):
> >round-trip min/avg/max/stddev = 2.273/15.264/497.869/44.018 ms . With
> >an average latency of 15ms, the time it took for  20MB file to transfer
> >over 9p roughly matches Floren's tests.
> >>
> >> I think that modifying 'cp' and 'exportfs' would resolve this problem
> >and will look at patching it per Floren's thesis to confirm.  Would
> >modifying 'cp' and 'exportfs' to allow for streams be something others
> >would be interested in? Or has this already been discussed and the
> >stream solution been found wanting for some technical reason? The
> >protocol, as mentioned in the paper, is called 9P2000.s. I don't see
> >that flavor listed here: http://9p.cat-v.org/implementations
> >>
> >> [1]
> >https://pdfs.semanticscholar.org/9b47/22e251df61b4abbe0da61720e57ac7b62f61.pdf
> >>
>

Re: [9front] Slow cp, support for 9P2000.s

[Jens Staal]

Den fre 24 apr. 2020 kl 13:36 skrev aiju <aiju@phicode.de>:
>
> None of the main team use reddit for Plan 9 stuff, afaik.
>

Too bad. Would be cool with a RedditFS and browse Reddit with Acme ...


> 9P2020 (fwiw, not the only name that's been used) has been discussed on
> IRC (freenode, #cat-v) extensively in the past.
>
> aiju
>

Re: [9front] Slow cp, support for 9P2000.s

[hiro]

i wouldn't waste any time trying to interface reddit, considering they
are so shameless about destroying the main interfaces people are using
on their website.
I.e. nowadays every reddit thread is unreadable unless you replace
www.reddit.com part of url with old.reddit.com (i do this every time,
manually, and i'm sick of it).
and that's not even a weird API, it is the *main* way of accessing
reddit that they stopped caring about for some reason...

On 4/24/20, Jens Staal <staal1978@gmail.com> wrote:
> Den fre 24 apr. 2020 kl 13:36 skrev aiju <aiju@phicode.de>:
>>
>> None of the main team use reddit for Plan 9 stuff, afaik.
>>
>
> Too bad. Would be cool with a RedditFS and browse Reddit with Acme ...
>
>
>> 9P2020 (fwiw, not the only name that's been used) has been discussed on
>> IRC (freenode, #cat-v) extensively in the past.
>>
>> aiju
>>
>

Re: [9front] Slow cp, support for 9P2000.s

[Jens Staal]

Den fre 24 apr. 2020 kl 14:13 skrev hiro <23hiro@gmail.com>:
>
> i wouldn't waste any time trying to interface reddit, considering they
> are so shameless about destroying the main interfaces people are using
> on their website.

The terminal viewer rtv is still working although I do not think it is
developed anymore.
https://github.com/michael-lazar/rtv

unfortunately built in python (and probably a bunch of dynamic dependencies).
I just saw that you can ssh to redditbox.us I need to check that out
on 9front ;)
Since the first thing that happens is a TUI menu for interface I guess
it will not work in a Plan9 terminal.

> I.e. nowadays every reddit thread is unreadable unless you replace
> www.reddit.com part of url with old.reddit.com (i do this every time,
> manually, and i'm sick of it).
> and that's not even a weird API, it is the *main* way of accessing
> reddit that they stopped caring about for some reason...
>

There is a fuse file system for reddit (also in python). So I guess
there is some sort of standardized way to access it independently.
https://github.com/redditvfs/redditvfs

> On 4/24/20, Jens Staal <staal1978@gmail.com> wrote:
> > Den fre 24 apr. 2020 kl 13:36 skrev aiju <aiju@phicode.de>:
> >>
> >> None of the main team use reddit for Plan 9 stuff, afaik.
> >>
> >
> > Too bad. Would be cool with a RedditFS and browse Reddit with Acme ...
> >
> >
> >> 9P2020 (fwiw, not the only name that's been used) has been discussed on
> >> IRC (freenode, #cat-v) extensively in the past.
> >>
> >> aiju
> >>
> >

Re: [9front] Slow cp, support for 9P2000.s

[rgl]

you have vt(1) to deal with that.

Re: [9front] Slow cp, support for 9P2000.s

[hiro]

and how long will it work? won't they change their api?

9P2000.s vs 9P2020 vs. ? (Was Re: [9front] Slow cp, support for 9P2000.s)

[Romano]

Since there's no existing discussion I can see, and IRC isn't meant for searching for answers to questions, is there anyone willing to summarize the state of solving problems like streaming for 9P?

If Aiju remembers correctly, there were two things where 9P2000.s was found wanting:
1) it breaks the syscall interface; and
2) it forces 9P to run over TCP.

FWIW, I finally finished reading through all of John's thesis, rather than just skimming, and I think John did a nice review of the previous work done in this area and presenting why he chose the solution he did.  His solution used TCP, but from his write-up, I don't see a technical reason why another transport protocol[1] could not be used. That would leave only breaking the syscall interface.

Does breaking the syscall interface mean that it's merely adding a syscall? Or does it mean something else?

Right now hget(1) is an rc script that uses webfs(4).  So if I'm at a terminal far away and am using webfs provided by the server I'm connected to, hget would be slow for a large file because the file would be transferred over 9P. It's no better than Plan 9's implementation, but no worse either.

[1] https://en.wikipedia.org/wiki/Category:Transport_layer_protocols

Re: 9P2000.s vs 9P2020 vs. ? (Was Re: [9front] Slow cp, support for 9P2000.s)

[Roman Shaposhnik]

On Wed, May 13, 2020 at 1:27 AM Romano <unobe@cpan.org> wrote:
>
> Since there's no existing discussion I can see, and IRC isn't meant for searching for answers to questions, is there anyone willing to summarize the state of solving problems like streaming for 9P?
>
> If Aiju remembers correctly, there were two things where 9P2000.s was found wanting:
> 1) it breaks the syscall interface; and
> 2) it forces 9P to run over TCP.
>
> FWIW, I finally finished reading through all of John's thesis, rather than just skimming, and I think John did a nice review of the previous work done in this area and presenting why he chose the solution he did.  His solution used TCP, but from his write-up, I don't see a technical reason why another transport protocol[1] could not be used. That would leave only breaking the syscall interface.

Any chance to see a link to John's thesis?

> Does breaking the syscall interface mean that it's merely adding a syscall? Or does it mean something else?
>
> Right now hget(1) is an rc script that uses webfs(4).  So if I'm at a terminal far away and am using webfs provided by the server I'm connected to, hget would be slow for a large file because the file would be transferred over 9P. It's no better than Plan 9's implementation, but no worse either.
>
> [1] https://en.wikipedia.org/wiki/Category:Transport_layer_protocols

Re: 9P2000.s vs 9P2020 vs. ? (Was Re: [9front] Slow cp, support for 9P2000.s)

[Romano]

https://pdfs.semanticscholar.org/9b47/22e251df61b4abbe0da61720e57ac7b62f61.pdf

On May 18, 2020 12:07:13 AM UTC, Roman Shaposhnik <roman@shaposhnik.org> wrote:
>On Wed, May 13, 2020 at 1:27 AM Romano <unobe@cpan.org> wrote:
>>
>> Since there's no existing discussion I can see, and IRC isn't meant
>for searching for answers to questions, is there anyone willing to
>summarize the state of solving problems like streaming for 9P?
>>
>> If Aiju remembers correctly, there were two things where 9P2000.s was
>found wanting:
>> 1) it breaks the syscall interface; and
>> 2) it forces 9P to run over TCP.
>>
>> FWIW, I finally finished reading through all of John's thesis, rather
>than just skimming, and I think John did a nice review of the previous
>work done in this area and presenting why he chose the solution he did.
>His solution used TCP, but from his write-up, I don't see a technical
>reason why another transport protocol[1] could not be used. That would
>leave only breaking the syscall interface.
>
>Any chance to see a link to John's thesis?
>
>> Does breaking the syscall interface mean that it's merely adding a
>syscall? Or does it mean something else?
>>
>> Right now hget(1) is an rc script that uses webfs(4).  So if I'm at a
>terminal far away and am using webfs provided by the server I'm
>connected to, hget would be slow for a large file because the file
>would be transferred over 9P. It's no better than Plan 9's
>implementation, but no worse either.
>>
>> [1] https://en.wikipedia.org/wiki/Category:Transport_layer_protocols

Re: 9P2000.s vs 9P2020 vs. ? (Was Re: [9front] Slow cp, support for 9P2000.s)

[ori]

> Since there's no existing discussion I can see, and IRC isn't meant for
> searching for answers to questions, is there anyone willing to summarize
> the state of solving problems like streaming for 9P?

Nothing that we're satisfied with at the moment, though there's
thoughts on providing early acks on writes, and io agents to do
bring at least some of the io logic closer to the data, instead
of shouting across the internet for every transaction.

> If Aiju remembers correctly, there were two things where 9P2000.s was found wanting:
> 1) it breaks the syscall interface; and
> 2) it forces 9P to run over TCP.
>
> FWIW, I finally finished reading through all of John's thesis,
> rather than just skimming, and I think John did a nice review of
> the previous work done in this area and presenting why he chose
> the solution he did.  His solution used TCP, but from his
> write-up, I don't see a technical reason why another transport
> protocol[1] could not be used.  That would leave only breaking
> the syscall interface.
> 
> Does breaking the syscall interface mean that it's merely adding a
> syscall? Or does it mean something else?

Yes -- any program that would want to take advantage of it needs
to be touched.  And would need heuristics: a 9p roundtrip is
still fewer roundtrips than a TCP handshake, so using it blindly
would slow things down when handling small files.  We also, by
design, make no distinction between remote and local files, so
there would need to be more guesswork here.

It also doesn't help with, for example, walking a directory
heirarchy, so things like browsing a mail archive over 9p will
remain painfully slow.

Finally, if we want an ad-hoc solution like this, we can do it
entirely in userspace, with a TCP server sitting on a remote
server, and a local 9p server that opens a TCP connection to
that server and streams the whole file, providing it locally.
I don't think this would be very hard to write, and would
probably give the benefits of streams, without needing any
changes to the existing code. (Thinking about this, it may
actually be worth writing.)

So. in short: This is a big hole in the abstractions provided
by 9p, and it doesn't solve the problem completely.

> Right now hget(1) is an rc script that uses webfs(4).  So if
> I'm at a terminal far away and am using webfs provided by the
> server I'm connected to, hget would be slow for a large file
> because the file would be transferred over 9P. It's no better
> than Plan 9's implementation, but no worse either.
> 
> [1] https://en.wikipedia.org/wiki/Category:Transport_layer_protocols

Re: 9P2000.s vs 9P2020 vs. ? (Was Re: [9front] Slow cp, support for 9P2000.s)

[hiro]

none of the solutions so far proposed properly extend the original
abstraction without *creating* new holes.

i don't think there's "a big hole" without our optimization attempts...
as an example, when we have streams via the network, we will fill
buffers and packets have to be dropped.

if everything is local ressource usage fairness is only provided by
our scheduling, if everything is remote, let's say using multiple
hops, this creates new distributed scheduling problems and a lot of
distributed unfairness.

right now we don't notice, but this is the real hole here, elegantly
avoided so far by making everything synchronous.

On 5/18/20, ori@eigenstate.org <ori@eigenstate.org> wrote:
>> Since there's no existing discussion I can see, and IRC isn't meant for
>> searching for answers to questions, is there anyone willing to summarize
>> the state of solving problems like streaming for 9P?
>
> Nothing that we're satisfied with at the moment, though there's
> thoughts on providing early acks on writes, and io agents to do
> bring at least some of the io logic closer to the data, instead
> of shouting across the internet for every transaction.
>
>> If Aiju remembers correctly, there were two things where 9P2000.s was
>> found wanting:
>> 1) it breaks the syscall interface; and
>> 2) it forces 9P to run over TCP.
>>
>> FWIW, I finally finished reading through all of John's thesis,
>> rather than just skimming, and I think John did a nice review of
>> the previous work done in this area and presenting why he chose
>> the solution he did.  His solution used TCP, but from his
>> write-up, I don't see a technical reason why another transport
>> protocol[1] could not be used.  That would leave only breaking
>> the syscall interface.
>>
>> Does breaking the syscall interface mean that it's merely adding a
>> syscall? Or does it mean something else?
>
> Yes -- any program that would want to take advantage of it needs
> to be touched.  And would need heuristics: a 9p roundtrip is
> still fewer roundtrips than a TCP handshake, so using it blindly
> would slow things down when handling small files.  We also, by
> design, make no distinction between remote and local files, so
> there would need to be more guesswork here.
>
> It also doesn't help with, for example, walking a directory
> heirarchy, so things like browsing a mail archive over 9p will
> remain painfully slow.
>
> Finally, if we want an ad-hoc solution like this, we can do it
> entirely in userspace, with a TCP server sitting on a remote
> server, and a local 9p server that opens a TCP connection to
> that server and streams the whole file, providing it locally.
> I don't think this would be very hard to write, and would
> probably give the benefits of streams, without needing any
> changes to the existing code. (Thinking about this, it may
> actually be worth writing.)
>
> So. in short: This is a big hole in the abstractions provided
> by 9p, and it doesn't solve the problem completely.
>
>> Right now hget(1) is an rc script that uses webfs(4).  So if
>> I'm at a terminal far away and am using webfs provided by the
>> server I'm connected to, hget would be slow for a large file
>> because the file would be transferred over 9P. It's no better
>> than Plan 9's implementation, but no worse either.
>>
>> [1] https://en.wikipedia.org/wiki/Category:Transport_layer_protocols
>
>