9fans - fans of the OS Plan 9 from Bell Labs
 help / color / mirror / Atom feed
* [9fans] Dual dialing/forking sessions to increase 9P throughput
@ 2020-12-29 23:50 cigar562hfsp952fans
  2020-12-30  8:12 ` [9fans] " joey
                   ` (3 more replies)
  0 siblings, 4 replies; 18+ messages in thread
From: cigar562hfsp952fans @ 2020-12-29 23:50 UTC (permalink / raw)
  To: 9fans

It's well-known that 9P has trouble transferring large files (high
volume/high bandwith) over high-latency networks, such as the Internet.
While sleeping, one night last week, I got an idea which might improve
throughput:

#define PSEUDOCODE ("this is psuedocode, not actual C")

#define OJUMBO   1<<n
#define JUMBORSZ 1<<16
#define JUMBOWSZ 1<<16

cpuinit() {
  whatever1();
  kernel.jumborsz = get_op(boot9pini, "jumborsz") || JUMBORSZ;
  kernel.jumbowsz = get_op(boot9pini, "jumbowsz") || JUMBOWSZ;
  whatever2();
}

open(fd, path, mode, flags) {
  chan = files[fd];
  chan.srv = NULL;
  chan.bulksrv = NULL;
  chan.rsize = 0;
  chan.wsize = 0;
  if (flags & ORD) {
    chan.rsize = (flags & OJUMBO) ? kernel.jumborsz : IOUNIT;
  }
  if (flags & OWR) {
    chan.wsize = (flags & OJUMBO) ? kernel.jumbowsz : IOUNIT;
  }
  blah();
}

dial(fd, server) {
  chan = files[fd];
  srv = parse_addr(server);
  nextsrv = copy_srv(srv);
  ver9p = "9P2000";

  switch (srv.proto) {
  case TCP:
    iosize = max(chan.rsize, chan.wsize);  
    init_9p_tcp(srv.addr, ver9p, iosize);
    srv.error && return srv.error;
    chan.srv = srv;
    if (chan.flags & OJUMBO) {
      // try to open a second connection with normal IOUNIT
      init_9p_tcp(nextsrv.addr, ver9p, IOUNIT);
      if (!nextsrv.error) {
        chan.bulksrv = srv;
        chan.srv = nextsrv;
        set_tcp_opts(chan.srv, OLATENCY);
        set_tcp_opts(chan.bulksrv, OTHROUGHPUT);
      }
    }
    ;;
  }
  whatever();
}

write(fd, buf, len) {
  chan = files[fd];
  !chan.open && return ECLOSED;
  !(chan.mode & OWR) && return EPERM;
  srv = (len > IOUNIT) ? chan.bulksrv : chan.srv;
  return write_9p(srv.conn, buf, len);
}

main() {
  fd = open(filename, ORD|OWR|OJUMBO);
  read(fd, buf, n);
  write(fd, buf, n);
  celebrate_newfound_speed();
}

The idea, basically, is to use an open flag (OJUMBO) to signal that two
connections to the same server should be attempted.  If a second
connection can be established, it is used for normal 9P transactions,
while the first connection is used for large ("jumbo") writes.  Of
course, this approach will only work if the server forks and accepts
multiple connections.  If the second connection cannot be established,
open() falls back to its customary behavior (with, perhaps, a larger
iosize, depending on the setting of OJUMBO).

There is, however, a very simple reason why this approach won't really
work: the fids for a file opened on one connection won't be recognized
by the server on the other connection.  I guess those are the kinds of
details you miss when you engineer software in your sleep.  :) The qids
would probably still be the same, assuming the same server answers both
connections.  But even that can't be guaranteed.

If it were possible to fork a 9P session off, onto another connection,
something like this could work.  But that introduces the possibility
that messages could arrive at the server or client out-of-order, which
would violate the 9P protocol.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M7768691deb99f4bd060052c8
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [9fans] Re: Dual dialing/forking sessions to increase 9P throughput
  2020-12-29 23:50 [9fans] Dual dialing/forking sessions to increase 9P throughput cigar562hfsp952fans
@ 2020-12-30  8:12 ` joey
  2021-01-03 23:51 ` [9fans] " Ethan Gardener
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 18+ messages in thread
From: joey @ 2020-12-30  8:12 UTC (permalink / raw)
  To: 9fans

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

Do you know if there has ever been a comprehensive evaluation of tuning parameters for 9P?  I am sure from my previous post that it is obvious I am on the newer side of Plan9 users.

I feel like part of it could be a configuration issue, that is to say specifying the gbe vs ether data type and setting -m to a 9k size.  Additionally, would it violate the 9P protocol if you chunked the data first (i.e. if you have a file with 256 kbytes and ran it over 4 connections with 64 kbytes each).  There is an overhead dx/dt that would consume the gains at some point but from a theoretical stand point is it possible?  Or more accurately, would such an approach violate 9p?

> celebrate_newfound_speed();
This is honestly phenomenal :)

>  switch (srv.proto) {
  case TCP:
  iosize = max(chan.rsize, chan.wsize);  
  init_9p_tcp(srv.addr, ver9p, iosize);
Again maybe this is ignorance but my understanding was that while Plan9 can support a lot of things running TCP (for the rest of the world) it supports and prefers to utilize IL/9P for such a connection.  TCP vis-a-vis re-transmission throttling is universally bad, so it might be a function more of TCP then of the Plan9 server.  I once had a dedicated 100G link between Dallas and Denver and it initially pre-tuned only had about 4G in bandwidth (yes, this is not a typo).  Some simple tuning (both Linux devices) got that up to 50G almost immediately.  But TCP was the transport of choice and we never got to the 100G level, there were just too many variables and getting close would knock the connection bandwidth way back.  We only had the link for a short time, so possibly this could have been worked out but my point is that really anything over 1G copper cables is non-trivial when TCP is involved.

~Joey
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M06a2dd85933dbb4fe106607c
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2837 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2020-12-29 23:50 [9fans] Dual dialing/forking sessions to increase 9P throughput cigar562hfsp952fans
  2020-12-30  8:12 ` [9fans] " joey
@ 2021-01-03 23:51 ` Ethan Gardener
  2021-01-04 10:55   ` Ole-Hjalmar Kristensen
  2021-01-05  3:41 ` Joe S
  2021-01-25 22:31 ` David Arroyo
  3 siblings, 1 reply; 18+ messages in thread
From: Ethan Gardener @ 2021-01-03 23:51 UTC (permalink / raw)
  To: 9fans

> The idea, basically, is to use an open flag (OJUMBO) to signal that two
> connections to the same server should be attempted.  

What's the advantage over fcp(1)? 9p can have numerous requests "in flight" at once to work around latency issues, but of all the user programs, fcp is probably the only one which takes advantage of this.

> If a second
> connection can be established, it is used for normal 9P transactions,
> while the first connection is used for large ("jumbo") writes.

How large is "jumbo"? I believe all the user programs have 8KB buffers at present; are you going to change them all?

I'm not negative about this; just raising the points.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M59ac782c0f58a0cc1eec8c10
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-03 23:51 ` [9fans] " Ethan Gardener
@ 2021-01-04 10:55   ` Ole-Hjalmar Kristensen
  2021-01-04 11:03     ` hiro
  0 siblings, 1 reply; 18+ messages in thread
From: Ole-Hjalmar Kristensen @ 2021-01-04 10:55 UTC (permalink / raw)
  To: 9fans

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

I agree with you that using the existing tag mechanism to keep multiple
requests in flight should be sufficient. I get the impression that this is
not readily supported by the higher level libraries, though.

As an aside, I seem to remember that John Floren sugegsed (and implemented)
changes to the 9P protocol making it more suitable for streaming media by
creating another TCP connection on the side.

On Mon, Jan 4, 2021 at 12:52 AM Ethan Gardener <eekee57@fastmail.fm> wrote:

> > The idea, basically, is to use an open flag (OJUMBO) to signal that two
> > connections to the same server should be attempted.
>
> What's the advantage over fcp(1)? 9p can have numerous requests "in
> flight" at once to work around latency issues, but of all the user
> programs, fcp is probably the only one which takes advantage of this.
>
> > If a second
> > connection can be established, it is used for normal 9P transactions,
> > while the first connection is used for large ("jumbo") writes.
> 
> How large is "jumbo"? I believe all the user programs have 8KB buffers at
> present; are you going to change them all?
> 
> I'm not negative about this; just raising the points.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M32ea06de21be8e55f94d0b3b
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2726 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-04 10:55   ` Ole-Hjalmar Kristensen
@ 2021-01-04 11:03     ` hiro
  2021-01-04 19:08       ` joe
  0 siblings, 1 reply; 18+ messages in thread
From: hiro @ 2021-01-04 11:03 UTC (permalink / raw)
  To: 9fans

reliability can be provided by AAN, already.

You can try out the Op protocol for inferno though, if you want to see
a working implementation of streaming 9p.
I've been running it for a long time and it worked perfectly for my use case.

On 1/4/21, Ole-Hjalmar Kristensen <ole.hjalmar.kristensen@gmail.com> wrote:
> I agree with you that using the existing tag mechanism to keep multiple
> requests in flight should be sufficient. I get the impression that this is
> not readily supported by the higher level libraries, though.
>
> As an aside, I seem to remember that John Floren sugegsed (and implemented)
> changes to the 9P protocol making it more suitable for streaming media by
> creating another TCP connection on the side.
>
> On Mon, Jan 4, 2021 at 12:52 AM Ethan Gardener <eekee57@fastmail.fm> wrote:
>
>> > The idea, basically, is to use an open flag (OJUMBO) to signal that two
>> > connections to the same server should be attempted.
>>
>> What's the advantage over fcp(1)? 9p can have numerous requests "in
>> flight" at once to work around latency issues, but of all the user
>> programs, fcp is probably the only one which takes advantage of this.
>>
>> > If a second
>> > connection can be established, it is used for normal 9P transactions,
>> > while the first connection is used for large ("jumbo") writes.
>>
>> How large is "jumbo"? I believe all the user programs have 8KB buffers at
>> present; are you going to change them all?
>>
>> I'm not negative about this; just raising the points.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M6f610222b3d7fc1800e82174
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-04 11:03     ` hiro
@ 2021-01-04 19:08       ` joe
  2021-01-04 20:44         ` hiro
  2021-01-04 21:01         ` ori
  0 siblings, 2 replies; 18+ messages in thread
From: joe @ 2021-01-04 19:08 UTC (permalink / raw)
  To: 9fans

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

Might be more than it is worth but what if you made a small program/file server that addressed portions of the "larger" file as their own file in a temporary directory. On write you would have to preallocate the file on the disk and write eatch section to it's respective range. "Chunks" would be simplified by having a pre-copy size and a "Chunk number" appended on the end of each chunk. Used to write to the correct location in the destination. Everything in else could remain the same. You would just use the "Split/Bulk Copy" program to copy from source to destination. Getting better with plan9 but not enough so to implement something like this or know if it would work. Just a thought.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-Mb8a0447d7180b423909022ff
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1454 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-04 19:08       ` joe
@ 2021-01-04 20:44         ` hiro
  2021-01-04 21:01         ` ori
  1 sibling, 0 replies; 18+ messages in thread
From: hiro @ 2021-01-04 20:44 UTC (permalink / raw)
  To: 9fans

that doesn't make any sense.
9p already allows you to access many parts of the file at the same time...

On 1/4/21, joe@lifesoftserv.com <joe@lifesoftserv.com> wrote:
> Might be more than it is worth but what if you made a small program/file
> server that addressed portions of the "larger" file as their own file in a
> temporary directory. On write you would have to preallocate the file on the
> disk and write eatch section to it's respective range. "Chunks" would be
> simplified by having a pre-copy size and a "Chunk number" appended on the
> end of each chunk. Used to write to the correct location in the destination.
> Everything in else could remain the same. You would just use the "Split/Bulk
> Copy" program to copy from source to destination. Getting better with plan9
> but not enough so to implement something like this or know if it would work.
> Just a thought.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M0af38d162f862329a4edc3c3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-04 19:08       ` joe
  2021-01-04 20:44         ` hiro
@ 2021-01-04 21:01         ` ori
  1 sibling, 0 replies; 18+ messages in thread
From: ori @ 2021-01-04 21:01 UTC (permalink / raw)
  To: 9fans

Quoth joe@lifesoftserv.com:
> Might be more than it is worth but what if you made a small
> program/file server that addressed portions of the "larger" file as
> their own file in a temporary directory.  On write you would have to
> preallocate the file on the disk and write eatch section to it's
> respective range.  "Chunks" would be simplified by having a pre-copy
> size and a "Chunk number" appended on the end of each chunk.  Used to
> write to the correct location in the destination.  Everything in else
> could remain the same.  You would just use the "Split/Bulk Copy"
> program to copy from source to destination.  Getting better with plan9
> but not enough so to implement something like this or know if it would
> work.  Just a thought.

http://man.9front.org/4/cfs

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-Mf9a01546241ea2a6777788c1
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2020-12-29 23:50 [9fans] Dual dialing/forking sessions to increase 9P throughput cigar562hfsp952fans
  2020-12-30  8:12 ` [9fans] " joey
  2021-01-03 23:51 ` [9fans] " Ethan Gardener
@ 2021-01-05  3:41 ` Joe S
  2021-01-05  9:40   ` hiro
                     ` (2 more replies)
  2021-01-25 22:31 ` David Arroyo
  3 siblings, 3 replies; 18+ messages in thread
From: Joe S @ 2021-01-05  3:41 UTC (permalink / raw)
  To: 9fans

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

hiro:
    The purpose of doing this was not to just access multiple parts of the file. I was working off the quote below.  Maybe a file server thats purpose is to mux parts of another file sounded like fun. My thoughts are that you could then transer thoes chunks on a single destination on seperate connections.

eg.
    % mux -C 3 -F large.file
    % tree.
          ├── large.file
          └── large.file.mux
              ├── 1
              ├── 2
              └── 3

1 directory, 4 files

Like I said though still learning though.

On Wednesday, 30 December 2020, at 12:20 AM, cigar562hfsp952fans wrote:
> There is, however, a very simple reason why this approach won't really
work: the fids for a file opened on one connection won't be recognized
by the server on the other connection.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-Mb319f4f5aae92c0b9de48f76
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2188 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-05  3:41 ` Joe S
@ 2021-01-05  9:40   ` hiro
  2021-01-06 22:21   ` [9fans] " cigar562hfsp952fans
  2021-01-07 16:02   ` [9fans] " ori
  2 siblings, 0 replies; 18+ messages in thread
From: hiro @ 2021-01-05  9:40 UTC (permalink / raw)
  To: 9fans

yes but there is no benefit to do that

On 1/5/21, Joe S <joe@lifesoftserv.com> wrote:
> hiro:
>     The purpose of doing this was not to just access multiple parts of the
> file. I was working off the quote below.  Maybe a file server thats purpose
> is to mux parts of another file sounded like fun. My thoughts are that you
> could then transer thoes chunks on a single destination on seperate
> connections.
>
> eg.
>     % mux -C 3 -F large.file
>     % tree.
>           ├── large.file
>           └── large.file.mux
>               ├── 1
>               ├── 2
>               └── 3
>
> 1 directory, 4 files
>
> Like I said though still learning though.
>
> On Wednesday, 30 December 2020, at 12:20 AM, cigar562hfsp952fans wrote:
>> There is, however, a very simple reason why this approach won't really
> work: the fids for a file opened on one connection won't be recognized
> by the server on the other connection.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M4e51600cccb05b53aabb5783
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* [9fans] Re: Dual dialing/forking sessions to increase 9P throughput
  2021-01-05  3:41 ` Joe S
  2021-01-05  9:40   ` hiro
@ 2021-01-06 22:21   ` cigar562hfsp952fans
  2021-01-07  8:06     ` hiro
  2021-01-07 16:02   ` [9fans] " ori
  2 siblings, 1 reply; 18+ messages in thread
From: cigar562hfsp952fans @ 2021-01-06 22:21 UTC (permalink / raw)
  To: 9fans

"Ethan Gardener" <eekee57@fastmail.fm> writes:

> What's the advantage over fcp(1)? 9p can have numerous requests "in
> flight" at once to work around latency issues, but of all the user
> programs, fcp is probably the only one which takes advantage of this.

The problem is that throughput and responsiveness are enemies.  If
you're trying to push a large quantity of data down a TCP connection,
and try to do anything interactive over that same connection, the
latency will make that interaction very slow.  In order to improve
responsiveness, the transport medium needs to know how to differentiate
between the two kinds of traffic.  In other words, you would need a way
to set different TCP options on 9P messages for different purposes.

"Joe S" <joe@lifesoftserv.com> writes:

> Maybe a file server thats purpose is to mux parts of another file
> sounded like fun. My thoughts are that you could then transer thoes
> chunks on a single destination on seperate connections.

I might be possible to create a file server which interposes itself
between a 9P client and 9P server and bonds multiple network connections
together into a single 9P session.  It would be sort of a userspace
replacement for the kernel's mount driver.  Wait a minute, this problem
can be solved with a filesystem?  Of course... I should have thought of
that.  ;)

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-Mdb27b326eb1a0019858aa910
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Re: Dual dialing/forking sessions to increase 9P throughput
  2021-01-06 22:21   ` [9fans] " cigar562hfsp952fans
@ 2021-01-07  8:06     ` hiro
  0 siblings, 0 replies; 18+ messages in thread
From: hiro @ 2021-01-07  8:06 UTC (permalink / raw)
  To: 9fans

> The problem is that throughput and responsiveness are enemies.
no. not throughput, but buffer sizes are the enemy of latency

> you're trying to push a large quantity of data down a TCP connection,
> and try to do anything interactive over that same connection, the
> latency will make that interaction very slow

no. you have full freedom to prioritize the interactive part on
application layer (since you mention TCP layer).

> the transport medium needs to know how to differentiate
> between the two kinds of traffic.
no, any other layer could also differentiate the traffic.

> you would need a way to set different TCP options on 9P messages for different purposes.

TCP doesn't differentiate well. TCP fairness is not fine-grained
enough for this, so i still don't think the transport layer is well
suited.

what we really need is 9p (file/fd) level distributed scheduling.

>
> "Joe S" <joe@lifesoftserv.com> writes:
>
>> Maybe a file server thats purpose is to mux parts of another file
>> sounded like fun. My thoughts are that you could then transer thoes
>> chunks on a single destination on seperate connections.
> 
> I might be possible to create a file server which interposes itself
> between a 9P client and 9P server and bonds multiple network connections
> together into a single 9P session.  It would be sort of a userspace
> replacement for the kernel's mount driver.  Wait a minute, this problem
> can be solved with a filesystem?  Of course... I should have thought of
> that.  ;)

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M8679a1e06d55c6877dbfbdfe
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-05  3:41 ` Joe S
  2021-01-05  9:40   ` hiro
  2021-01-06 22:21   ` [9fans] " cigar562hfsp952fans
@ 2021-01-07 16:02   ` ori
  2 siblings, 0 replies; 18+ messages in thread
From: ori @ 2021-01-07 16:02 UTC (permalink / raw)
  To: 9fans

Quoth Joe S <joe@lifesoftserv.com>:
> hiro:
>     The purpose of doing this was not to just access multiple parts of the file. I was working off the quote below.  Maybe a file server thats purpose is to mux parts of another file sounded like fun. My thoughts are that you could then transer thoes chunks on a single destination on seperate connections.
> 
> eg.
>     % mux -C 3 -F large.file
>     % tree.
>           ├── large.file
>           └── large.file.mux
>               ├── 1
>               ├── 2
>               └── 3
> 
> 1 directory, 4 files
> 
> Like I said though still learning though.

You don't need separate connections to transfer
chunks in parallel, you need multiple Tread
messages in flight at once. The only two ways
to do that on plan 9 is pread() from multiple
threads(procs) at once, either in an external
tool like fcp(1), or in the program itself.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M5f334f5b561aae6155a06c16
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2020-12-29 23:50 [9fans] Dual dialing/forking sessions to increase 9P throughput cigar562hfsp952fans
                   ` (2 preceding siblings ...)
  2021-01-05  3:41 ` Joe S
@ 2021-01-25 22:31 ` David Arroyo
  2021-01-27 16:25   ` Ethan Gardener
  2021-01-27 16:52   ` ori
  3 siblings, 2 replies; 18+ messages in thread
From: David Arroyo @ 2021-01-25 22:31 UTC (permalink / raw)
  To: Kyle Farwell via 9fans

On Tue, Dec 29, 2020, at 18:50, cigar562hfsp952fans@icebubble.org wrote:
> It's well-known that 9P has trouble transferring large files (high
> volume/high bandwith) over high-latency networks, such as the Internet.

From what I know of 9P, I don't think this is the fault of the protocol
itself. In fact, since 9P lets the clients choose Fid and Tag identifiers,
it should be uniquely well suited for "long fat pipes". You could avoid
waiting for round-trips by optimistically assuming your requests succeed.
For example, you could do the following to optimistically read the first
8K bytes of a file without needing to wait for a response from the server.

* Twalk tag=1 fid=0 newfid=1 /path/to/somefile
* Topen tag=2 fid=1 o_read
* Tread tag=3 fid=1 off=0 count=4096
* Tread tag=4 fid=1 off=4096 count=4096
* Tclunk tag=5 fid=1

I'm not aware of any client implementations that do this kind of
pipelining, though.

David

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M108d4f8ab0862aef8c9c4202
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-25 22:31 ` David Arroyo
@ 2021-01-27 16:25   ` Ethan Gardener
  2021-01-27 16:52   ` ori
  1 sibling, 0 replies; 18+ messages in thread
From: Ethan Gardener @ 2021-01-27 16:25 UTC (permalink / raw)
  To: 9fans

On Mon, Jan 25, 2021, at 10:31 PM, David Arroyo wrote:
> On Tue, Dec 29, 2020, at 18:50, cigar562hfsp952fans@icebubble.org wrote:
> > It's well-known that 9P has trouble transferring large files (high
> > volume/high bandwith) over high-latency networks, such as the Internet.
> 
> From what I know of 9P, I don't think this is the fault of the protocol
> itself. In fact, since 9P lets the clients choose Fid and Tag identifiers,
> it should be uniquely well suited for "long fat pipes". You could avoid
> waiting for round-trips by optimistically assuming your requests succeed.
> For example, you could do the following to optimistically read the first
> 8K bytes of a file without needing to wait for a response from the server.
> 
> * Twalk tag=1 fid=0 newfid=1 /path/to/somefile
> * Topen tag=2 fid=1 o_read
> * Tread tag=3 fid=1 off=0 count=4096
> * Tread tag=4 fid=1 off=4096 count=4096
> * Tclunk tag=5 fid=1
> 
> I'm not aware of any client implementations that do this kind of
> pipelining, though.

fcp(1)? 

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M1b4dd6e347d2d00d2cd645d8
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-25 22:31 ` David Arroyo
  2021-01-27 16:25   ` Ethan Gardener
@ 2021-01-27 16:52   ` ori
  2021-01-27 17:34     ` Charles Forsyth
  2021-01-28  0:17     ` David Arroyo
  1 sibling, 2 replies; 18+ messages in thread
From: ori @ 2021-01-27 16:52 UTC (permalink / raw)
  To: 9fans

Quoth David Arroyo <droyo@aqwari.net>:
> On Tue, Dec 29, 2020, at 18:50, cigar562hfsp952fans@icebubble.org wrote:
> > It's well-known that 9P has trouble transferring large files (high
> > volume/high bandwith) over high-latency networks, such as the Internet.
> 
> From what I know of 9P, I don't think this is the fault of the protocol
> itself. In fact, since 9P lets the clients choose Fid and Tag identifiers,
> it should be uniquely well suited for "long fat pipes". You could avoid
> waiting for round-trips by optimistically assuming your requests succeed.
> For example, you could do the following to optimistically read the first
> 8K bytes of a file without needing to wait for a response from the server.
> 
> * Twalk tag=1 fid=0 newfid=1 /path/to/somefile
> * Topen tag=2 fid=1 o_read
> * Tread tag=3 fid=1 off=0 count=4096
> * Tread tag=4 fid=1 off=4096 count=4096
> * Tclunk tag=5 fid=1
> 
> I'm not aware of any client implementations that do this kind of
> pipelining, though.
> 
> David

This also has some hairy edge cases. For example,
what happens to the actions in the pipeline if one
of the operations fails?

I think that for this kind of pipelining to be
effective, 9p may need some notion of 'bundles',
where the first failure, short write, or other
exceptional operation would cause all other
commands in the bundle to be ignored.

Another issue is that its' difficult to retrofit
this pipelining into the plan 9 system call
interface; when do you return an error from
read(2)? what if there are mutiple Treads?

Finally, there's the problem of flow control;
with 9p, round trip time limits starvation,
at least to a degree. But how do you handle
congestion if you can stuff as many 9p packets
down a single connection as possible? There's
no packet loss, but you can end up with very
long delays as small reads like /dev/mouse
get queued behind bulk transfers.

There are some programs that try to handle this
for 'well behaved' file systems (eg, fcp(1),
or http://src.a-b.xyz/clone/, but making it
happen in general is not trivial.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-Maff5e0961941062961ff4f2d
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-27 16:52   ` ori
@ 2021-01-27 17:34     ` Charles Forsyth
  2021-01-28  0:17     ` David Arroyo
  1 sibling, 0 replies; 18+ messages in thread
From: Charles Forsyth @ 2021-01-27 17:34 UTC (permalink / raw)
  To: 9fans

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

I found several of Bryan Ford's ideas interesting as always here,
particularly Structured Streams (https://bford.info/pub/net/sst-abs/) and
Breaking Up the Transport Logjam (https://bford.info/pub/net/logjam-abs/)

On Wed, Jan 27, 2021 at 4:53 PM <ori@eigenstate.org> wrote:

> Quoth David Arroyo <droyo@aqwari.net>:
> > On Tue, Dec 29, 2020, at 18:50, cigar562hfsp952fans@icebubble.org wrote:
> > > It's well-known that 9P has trouble transferring large files (high
> > > volume/high bandwith) over high-latency networks, such as the Internet.
> >
> > From what I know of 9P, I don't think this is the fault of the protocol
> > itself. In fact, since 9P lets the clients choose Fid and Tag
> identifiers,
> > it should be uniquely well suited for "long fat pipes". You could avoid
> > waiting for round-trips by optimistically assuming your requests succeed.
> > For example, you could do the following to optimistically read the first
> > 8K bytes of a file without needing to wait for a response from the
> server.
> >
> > * Twalk tag=1 fid=0 newfid=1 /path/to/somefile
> > * Topen tag=2 fid=1 o_read
> > * Tread tag=3 fid=1 off=0 count=4096
> > * Tread tag=4 fid=1 off=4096 count=4096
> > * Tclunk tag=5 fid=1
> >
> > I'm not aware of any client implementations that do this kind of
> > pipelining, though.
> >
> > David
> 
> This also has some hairy edge cases. For example,
> what happens to the actions in the pipeline if one
> of the operations fails?
> 
> I think that for this kind of pipelining to be
> effective, 9p may need some notion of 'bundles',
> where the first failure, short write, or other
> exceptional operation would cause all other
> commands in the bundle to be ignored.
> 
> Another issue is that its' difficult to retrofit
> this pipelining into the plan 9 system call
> interface; when do you return an error from
> read(2)? what if there are mutiple Treads?
> 
> Finally, there's the problem of flow control;
> with 9p, round trip time limits starvation,
> at least to a degree. But how do you handle
> congestion if you can stuff as many 9p packets
> down a single connection as possible? There's
> no packet loss, but you can end up with very
> long delays as small reads like /dev/mouse
> get queued behind bulk transfers.
> 
> There are some programs that try to handle this
> for 'well behaved' file systems (eg, fcp(1),
> or http://src.a-b.xyz/clone/, but making it
> happen in general is not trivial.
> 

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M40bb94bd40d97df72c885f69
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 4542 bytes --]

^ permalink raw reply	[flat|nested] 18+ messages in thread

* Re: [9fans] Dual dialing/forking sessions to increase 9P throughput
  2021-01-27 16:52   ` ori
  2021-01-27 17:34     ` Charles Forsyth
@ 2021-01-28  0:17     ` David Arroyo
  1 sibling, 0 replies; 18+ messages in thread
From: David Arroyo @ 2021-01-28  0:17 UTC (permalink / raw)
  To: Kyle Farwell via 9fans

On Wed, Jan 27, 2021, at 11:25, Ethan Gardener wrote:
> fcp(1)? 

You are right, but fcp(1) would only produce multiple parallel read
and write messages, I'm talking about a more general approach. fcp is
probably a better compromise, though, as ori points out the client can
get complicated pretty quickly when trying to solve the problem more
generally:

On Wed, Jan 27, 2021, at 11:52, ori@eigenstate.org wrote:
> This also has some hairy edge cases. For example,
> what happens to the actions in the pipeline if one
> of the operations fails?

Yes, the tradeoff of this pipelining is that the client becomes a lot
more complex, but for this specific problem, the client would need to be
prepared to receive and discard Rerror messages for each message after
the failed one, just surfacing the first error to whatever system call
or library function kicked off this sequence of messages.

> I think that for this kind of pipelining to be
> effective, 9p may need some notion of 'bundles',
> where the first failure, short write, or other
> exceptional operation would cause all other
> commands in the bundle to be ignored.

You can get pretty close to "bundles" without changing the protocol by
having the client reserve a few bits of storage in the Tag header to
group requests in a pipeline together:

# Bundle 1, sent at once
Twalk tag=0x0100 fid=0 newfid=1 "foo/bar"
Topen tag=0x0101 fid=1 O_RDONLY
Twrite tag=0x0102 fid=1 data
Tclunk tag=0x0103 fid=1

Then the client could use a trie or some similar data structure on the
tag of an R-message to get the "state" of a pipeline.

> Another issue is that its' difficult to retrofit
> this pipelining into the plan 9 system call
> interface; when do you return an error from
> read(2)? what if there are mutiple Treads?

This is a harder problem, I think. What if non-contiguous reads or writes
succeed? An mmap()-style API might fare better here, but that comes with
its own drawbacks. I don't have a good answer here, but I think since
Twrite and Rread messages are elastic it is always better to just send
larger messages, increasing msize if necessary.

> how do you handle congestion if you can stuff as many 9p packets down
> a single connection as possible? There's no packet loss, but you can
> end up with very long delays as small reads like /dev/mouse get queued
> behind bulk transfers.

The problem you describe is analagous to the "buffer bloat" problem.
In the lower protocols like TCP it is solved by making the sender aware of
the bandwidth delay product and sizing its buffers appropriately. So it
could be handled by using a model-based congestion avoidance algorithm
like BBR or Vegas, and sending messages out of a prioritized queue,
where "interactive" files like /dev/mouse are prioritized.

David

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te69bb0fce0f0ffaf-M6e7a64ad95ea27ac3c73a9ac
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2021-01-28  0:18 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-29 23:50 [9fans] Dual dialing/forking sessions to increase 9P throughput cigar562hfsp952fans
2020-12-30  8:12 ` [9fans] " joey
2021-01-03 23:51 ` [9fans] " Ethan Gardener
2021-01-04 10:55   ` Ole-Hjalmar Kristensen
2021-01-04 11:03     ` hiro
2021-01-04 19:08       ` joe
2021-01-04 20:44         ` hiro
2021-01-04 21:01         ` ori
2021-01-05  3:41 ` Joe S
2021-01-05  9:40   ` hiro
2021-01-06 22:21   ` [9fans] " cigar562hfsp952fans
2021-01-07  8:06     ` hiro
2021-01-07 16:02   ` [9fans] " ori
2021-01-25 22:31 ` David Arroyo
2021-01-27 16:25   ` Ethan Gardener
2021-01-27 16:52   ` ori
2021-01-27 17:34     ` Charles Forsyth
2021-01-28  0:17     ` David Arroyo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).