Re: [9fans] 9p over high-latency

Re: [9fans] 9p over high-latency

[sqweek]

On Thu, Sep 18, 2008 at 7:51 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>>  Hm, but what's the alternative here? Readahead seems somewhat
>> attractive, if difficult (I worry about blocking reads and timing
>> sensitive file systems).
>
> the fundamental problem is that it becomes very difficult to
> implement fileservers which don't serve up regular files.
> you might make perminant changes to something stored on
> a disk with readahead.

 I'm not sure this problem is insurmountable. Some simple heuristics
along the lines of waiting a couple of Rreads to see if it looks like
an i/o pattern that is slurping up the whole file could go a long way.
However, I agree this is not the most reliable approach, and certainly
doesn't fit with the simplicity ideal.

> i'll admit that i don't understand the point of batching walks.
> i'm not sure why one would set up a case where you know you'll
> have a long network and where you know you'll need to execute
> a lot of walks.  most applications that do most i/o in a particular
> directory set . to that directory to avoid the walks.

 Right, but unless the mnt driver keeps an fid around for every file
in that directory set, every open is still two round trips - one to
walk to the file, and one to actually open it. Which is almost twice
as long as necessary. You might scoff at a single round trip, but as
someone who uses ircfs over the internet, I can tell you the
difference in responsiveness between echo and cat is noticeable
(though ok, echo has two extra round trips in this case).

> i'm not sure that octopus wouldn't be better off optimizing
> latency by running many more threads.  but that's just an ignorant
> opinion.

 How can multiple threads possibly help with latency caused by
operations that forced to be serial by the protocol? In fact, how can
multithreading help with latency in general? It will help
*throughput*, certainly, but unless you have a thread dedicated to
predicting the future, I can't see where the benifit will come from.
-sqweek

Re: [9fans] 9p over high-latency

[Eric Van Hensbergen]

On Thu, Sep 18, 2008 at 8:34 AM, sqweek <sqweek@gmail.com> wrote:
> On Thu, Sep 18, 2008 at 7:51 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>
>  How can multiple threads possibly help with latency caused by
> operations that forced to be serial by the protocol? In fact, how can
> multithreading help with latency in general? It will help
> *throughput*, certainly, but unless you have a thread dedicated to
> predicting the future, I can't see where the benifit will come from.
>

I'm not saying I know it would be a good idea, but you could implement
speculative-9P which issued the equivalent of the batched requests
without waiting for the responses of the prior request -- since you
are on an in-order pipe the file server would get the requests in
order and if they all succeeded, then you've reduced some latency.
Of course any failure will cause issues, potentially bad issues (a
bunch of walks followed by a create would end badly if one of the
walks failed and the create put the file someplace undesirable).

I think this is tending towards what was discussed at the first IWP9
-- IIRC the idea was to use the same tid for all the operations
(instead of a new tid per operation as would normally be done in
multi-threaded 9P).  An error or short-walk on any of the ops
associated with that tid.  I can't remember if we needed some op to
represent transaction boundries in order to recover failed tids --
anyone's memory better than mine?

The problem again here is added complexity to client and server who
have to track more state associated with these transactions....

               -eric

Re: [9fans] 9p over high-latency

[Francisco J Ballesteros]

> I'm not saying I know it would be a good idea, but you could implement
> speculative-9P which issued the equivalent of the batched requests
> without waiting for the responses of the prior request -- since you
> are on an in-order pipe the file server would get the requests in
> order and if they all succeeded, then you've reduced some latency.

We did this, IIRC. It helped reduce latency, but for file trees like
the one used
by omero it was not enough (at least for 150ms rtts). In the end, the
file server
is pushing changes to the viewer via a "connection" file.

Of course, for other devices it might be fine.

Re: [9fans] 9p over high-latency

[Steve Simon]

> We did this, IIRC. It helped reduce latency, but...

Are the modified kernel files still in your venti?
I would be interested in having a play in a simple disk file
sharing environment.

-Steve

Re: [9fans] 9p over high-latency

[Francisco J Ballesteros]

I think I made it using a user-level program to talk to 9p
servers trying to do some caching. I'll try to find out if I still
have it in venti,
I think I just removed
the thing after finding out it was not going to work for retrieving omero trees.

On Thu, Sep 18, 2008 at 10:05 PM, Steve Simon <steve@quintile.net> wrote:
>> We did this, IIRC. It helped reduce latency, but...
>
> Are the modified kernel files still in your venti?
> I would be interested in having a play in a simple disk file
> sharing environment.
>
> -Steve
>
>

Re: [9fans] 9p over high-latency

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

On Thu, Sep 18, 2008 at 05:57:21PM +0800, sqweek wrote:
> On Fri, Sep 12, 2008 at 7:47 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> > as an aside: i don't think 9p itself limits plan 9 performance
> > over high-latency links.  the limitations have more to do with
> > the number of outstanding messages, which is 1 in the mnt
> > driver.
> 
>  Hm, but what's the alternative here? Readahead seems somewhat
> attractive, if difficult (I worry about blocking reads and timing
> sensitive file systems). But there's one problem I can't resolve - how
> do you know what offset to Tread without consulting the previous
> Rread's count?
>  Actually, I understand there has been discussion about grouping tags
> to allow for things like Twalk/Topen batching without waiting for
> Rwalk (which sounds like a great idea), maybe that would work here
> also...
> -sqweek
> 

This seems an appropriate time -- well, appropriate thread, if a few days
lagged -- to bring up Dave's/my cache journaling scheme for review... I
presented this at IWP9 last year and recieved mostly positive feedback, as I
remember, but sadly have not had a chance to work on it since.  (That said,
it remains on my todo list should nobody beat me to it.)

The protocol design is documented on my server at
https://wiki.ietfng.org/pub/Plan9/JournalCallbacks.  Sadly, it remains the
case that I don't have an implementation to offer.

The basic idea is to shim a client-side cache and a server-side cache
controller in front of a filesystem, like fossil or exportfs, and have the
cache controller snoop on all connections to the server, informing caches
when the contents of their caches have become invalid.  There are a few
known deficiencies ([in]coherency and authentication being the two big
ones), but I hope the idea remains sound.

Thoughts and criticisms welcome. :)
--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] 9p over high-latency

[Eric Van Hensbergen]

On Thu, Sep 18, 2008 at 6:51 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> On Fri, Sep 12, 2008 at 7:47 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> also...
>
> the fundamental problem is that it becomes very difficult to
> implement fileservers which don't serve up regular files.
> you might make perminant changes to something stored on
> a disk with readahead.
>

My experience is that there are a couple of different scenarios here
-- there's dealing with synthetic file systems, dealing with regular
files, and then there is dealing with both.  Latency can effect all
three situations -- my understanding was that Op was actually
developed to deal with latency problems in dealing with the deep
hierarchies of the Octopus synthetic file systems.

There are likely a number of optimizations possible when dealing with
regular files -- but we currently don't give many/any hints in the
protocol as to what kind of optimizations are valid on a particular
fid -- and with things like batching walk/open its even more difficult
as you may cross mount points which invalidate the type of
optimization you think you can do.

Of course, if these were dealt with in a single protocol one approach
would be to return Error when attempting an invalid optimization
allowing clients to fall-back to a safer set of operations.  I do tend
to agree with Uriel that extensions, such as Op, may be better done in
a complimentary op-code space to make this sort of negotitation
possible.  Unfortunately this can add quite a bit of complexity to the
client and servers, so its not clear to me that its a definite win.

If you know you are dealing exclusively with regular files, I would
suggest starting with something like cfs(4) and play with different
potential optimizations there such as read-ahead, loose caches,
directory caches, temporal caches, etc.  Most of these techniques are
things you'd never want to look at with a synthetic file service, but
should provide a route for most of the optimizations you might want in
a wide-area-file-system -- particularly if you have exclusive access
and aren't worried about coherency.

If you are worried about coherency, you probably don't want to be
doing any of these optimizations.  There have been some conversations
about how to approach coherent caching, and I think some folks have
started working on it, but nothing is available yet.

                  -eric

Re: [9fans] 9p over high-latency

[erik quanstrom]

> On Fri, Sep 12, 2008 at 7:47 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> > as an aside: i don't think 9p itself limits plan 9 performance
> > over high-latency links.  the limitations have more to do with
> > the number of outstanding messages, which is 1 in the mnt
> > driver.
>
>  Hm, but what's the alternative here? Readahead seems somewhat
> attractive, if difficult (I worry about blocking reads and timing
> sensitive file systems). But there's one problem I can't resolve - how
> do you know what offset to Tread without consulting the previous
> Rread's count?
>  Actually, I understand there has been discussion about grouping tags
> to allow for things like Twalk/Topen batching without waiting for
> Rwalk (which sounds like a great idea), maybe that would work here
> also...

the fundamental problem is that it becomes very difficult to
implement fileservers which don't serve up regular files.
you might make perminant changes to something stored on
a disk with readahead.

since one of the main points of plan 9 is to get rid of special
files, ioctl's and whatnot, read ahead seems unattactive.

i'll admit that i don't understand the point of batching walks.
i'm not sure why one would set up a case where you know you'll
have a long network and where you know you'll need to execute
a lot of walks.  most applications that do most i/o in a particular
directory set . to that directory to avoid the walks.

i'm not sure that octopus wouldn't be better off optimizing
latency by running many more threads.  but that's just an ignorant
opinion.

- erik

Re: [9fans] 9p over high-latency

[Uriel]

Actually, this whole subject was thoroughly discussed at the *first*
iwp9; but apparently nobody is bothered by this enough to implement
any of the then suggested solutions.

Op is not one the solutions discussed at the time, and saying 'just
use another protocol' is not a way to resolve the shortcomings 9P
clearly has.

Moreover most of the solutions discussed back then seemed quite clear
and would be (mostly) backwards compatible, so I still don't
understand why Op was created rather than implement one of those
solutions, although my understanding is that nemo and co. considered
it easier (ie., didn't involve changing other people's code?).

Also worth noting is that Op has some serious shortcomings, if I
recall correctly it fails to translate some important 9P semantics
like walks, which means that while Op can transparently proxy 9P
connections, certain file servers wont work properly (I might be
mistaken, and certainly don't remember the details).

This is not to say that there is no use for a protocol like Op (it
shares much in common with http, which clearly has found lots of use
in the 'real world').

Peace

uriel

On Thu, Sep 18, 2008 at 12:36 PM, Christian Kellermann
<Christian.Kellermann@nefkom.net> wrote:
> * sqweek <sqweek@gmail.com> [080918 12:02]:
>> On Fri, Sep 12, 2008 at 7:47 PM, erik quanstrom <quanstro@quanstro.net> wrote:
>> > as an aside: i don't think 9p itself limits plan 9 performance
>> > over high-latency links.  the limitations have more to do with
>> > the number of outstanding messages, which is 1 in the mnt
>> > driver.
>>
>>  Hm, but what's the alternative here? Readahead seems somewhat
>> attractive, if difficult (I worry about blocking reads and timing
>> sensitive file systems). But there's one problem I can't resolve - how
>> do you know what offset to Tread without consulting the previous
>> Rread's count?
>>  Actually, I understand there has been discussion about grouping tags
>> to allow for things like Twalk/Topen batching without waiting for
>> Rwalk (which sounds like a great idea), maybe that would work here
>> also...
>
> There are some interesting approaches which have been discussed at the last iwp9, including op from nemo's team.
> http://plan9.bell-labs.com/iwp9/papers/10.op.esoriano.pdf
>
> Maybe that is worth looking at for your issue?
>
> Kind regards,
>
> Christian
>
> --
> You may use my gpg key for replies:
> pub  1024D/47F79788 2005/02/02 Christian Kellermann (C-Keen)
>

Re: [9fans] 9p over high-latency

[Christian Kellermann]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

* sqweek <sqweek@gmail.com> [080918 12:02]:
> On Fri, Sep 12, 2008 at 7:47 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> > as an aside: i don't think 9p itself limits plan 9 performance
> > over high-latency links.  the limitations have more to do with
> > the number of outstanding messages, which is 1 in the mnt
> > driver.
> 
>  Hm, but what's the alternative here? Readahead seems somewhat
> attractive, if difficult (I worry about blocking reads and timing
> sensitive file systems). But there's one problem I can't resolve - how
> do you know what offset to Tread without consulting the previous
> Rread's count?
>  Actually, I understand there has been discussion about grouping tags
> to allow for things like Twalk/Topen batching without waiting for
> Rwalk (which sounds like a great idea), maybe that would work here
> also...

There are some interesting approaches which have been discussed at the last iwp9, including op from nemo's team.
http://plan9.bell-labs.com/iwp9/papers/10.op.esoriano.pdf

Maybe that is worth looking at for your issue?

Kind regards,

Christian

-- 
You may use my gpg key for replies:
pub  1024D/47F79788 2005/02/02 Christian Kellermann (C-Keen)

[-- Attachment #2: Type: application/pgp-signature, Size: 202 bytes --]

Re: [9fans] 9p over high-latency

[sqweek]

On Fri, Sep 12, 2008 at 7:47 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> as an aside: i don't think 9p itself limits plan 9 performance
> over high-latency links.  the limitations have more to do with
> the number of outstanding messages, which is 1 in the mnt
> driver.

 Hm, but what's the alternative here? Readahead seems somewhat
attractive, if difficult (I worry about blocking reads and timing
sensitive file systems). But there's one problem I can't resolve - how
do you know what offset to Tread without consulting the previous
Rread's count?
 Actually, I understand there has been discussion about grouping tags
to allow for things like Twalk/Topen batching without waiting for
Rwalk (which sounds like a great idea), maybe that would work here
also...
-sqweek