[9fans] About 9P ...

[9fans] About 9P ...

[Philippe Anel]

Hello 9fans,

Regarding 9P design, I wonder why Fids are chosen by the client :
I tried to find an answer in the docs, manuals or in 9fans archive
but I have not found an answer (please tell me where I can find
an answer if I missed something).
Can someone explain me the reason why the Fids are not returned
by the server (along with the Qids by example) ?
Because Fids are chosen by the client, the server must search in
a table (hash table in lib9p) the associated file structure, Whereas
if Fids are chosen by the server, a simple array is enough. I don't
know however, if the performance improvement is really important.
In fact I'm quite sure it is not.
I also think this is not because of a security issue even if there
may be multiple clients sharing the same connection. Indeed, because
the transactions are only between the server and the kernel or an
remote (tcp/il) connection, it seems a process cannot steal the fid
of another process (or another namespace). Regarding the transactions
with the kernel, a process has no access to fids. And regarding
the transaction with a remote connection, the connection point can
be used as a kind of fid space.
Maybe I'm wrong about the security issue, I only want to know why
9P has been designed with such policy.

Thank you,
Phil;

Re: [9fans] About 9P ...

[Russ Cox]

> Because Fids are chosen by the client, the server must search in
> a table (hash table in lib9p) the associated file structure, Whereas
> if Fids are chosen by the server, a simple array is enough. I don't
> know however, if the performance improvement is really important.
> In fact I'm quite sure it is not.

If Fids were chosen by the server you would have written
the opposite.  One side can have a simple array lookup
and one side can't.  It doesn't matter either way.
You just pick one.

Or you could do what SSH did and pick two: there the
equivalent of a "fid" is actually a pair of integers,
one chosen by the client and one chosen by the server,
so that both can use an array lookup.  Because after
all the encryption and public key operations, it's really
looking up the fids that is eating all the cpu.

Russ

Re: [9fans] About 9P ...

[Roman Shaposhnick]

On Tue, 2007-06-19 at 20:37 -0400, Russ Cox wrote:
> Because after
> all the encryption and public key operations, it's really
> looking up the fids that is eating all the cpu.

  Now this is an interesting point, because it sort of argues
in favor of doing the lookups on the client and conserving
CPU cycles of the server where the cycles can be more precious.

Thanks,
Roman.

P.S. But then again, in Plan 9: the server is a client is a
server ;-)

Re: [9fans] About 9P ...

[Anthony Sorace]

perhaps my sense of humor's tweaked, but i understood russ' comment to
be sarcastic, since the added complexity of making fid lookup slightly
faster on both ends seems hard to justify given how vastly outweighed
fid lookup is bound to be by the encryption-related operations.

Re: [9fans] About 9P ...

[Derek Fawcus]

On Tue, Jun 19, 2007 at 10:11:31PM +0200, Philippe Anel wrote:
>
> Can someone explain me the reason why the Fids are not returned
> by the server (along with the Qids by example) ?

I always assumed it was to enable pipelined operations.

DF

Re: [9fans] About 9P ...

[Philippe Anel]

What do you mean ?

Derek Fawcus a écrit :
> On Tue, Jun 19, 2007 at 10:11:31PM +0200, Philippe Anel wrote:
>   
>> Can someone explain me the reason why the Fids are not returned
>> by the server (along with the Qids by example) ?
>>     
>
> I always assumed it was to enable pipelined operations.
>
> DF
>
>
>   

Re: [9fans] About 9P ...

[Derek Fawcus]

On Fri, Jun 22, 2007 at 08:58:04AM +0200, Philippe Anel wrote:
> What do you mean ?

Well,  I've not read the protocol details for a while.
But from memory I thought it allowed this type of operation:

    send: open,fid,file
    send: read,fid,args
    send: read,fid,args

    (wait one rtt)

    recv: open success/fail
    recv: read result / read error due to unknown fid
    recv: read result / read error due to unknown fid

DF

> 
> Derek Fawcus a écrit :
> > On Tue, Jun 19, 2007 at 10:11:31PM +0200, Philippe Anel wrote:
> >   
> >> Can someone explain me the reason why the Fids are not returned
> >> by the server (along with the Qids by example) ?
> >>     
> >
> > I always assumed it was to enable pipelined operations.
> >
> > DF
> >
> >
> >   

Re: [9fans] About 9P ...

[Philippe Anel]

Well even if it can work, I don't think 9P was designed to allow such
type of
operation. Indeed, 9P basis is the transaction : ie a Request followed
by a Reply.
So I think you should not pipeline the requests.
Can a 9P specialist can confirm this ? If so ... I now understand why
Fids are
(or must be) choosen by the client.

    Phil;
> Well,  I've not read the protocol details for a while.
> But from memory I thought it allowed this type of operation:
>
>     send: open,fid,file
>     send: read,fid,args
>     send: read,fid,args
>
>     (wait one rtt)
>
>     recv: open success/fail
>     recv: read result / read error due to unknown fid
>     recv: read result / read error due to unknown fid
>
> DF
>
>

Re: [9fans] About 9P ...

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

Transactions are tagged right?  So you can, in fact, have many in flight at
once.
Perhaps I'm missing what you meant by pipelined.


On 6/22/07, Philippe Anel <xigh@free.fr> wrote:
>
> Well even if it can work, I don't think 9P was designed to allow such
> type of
> operation. Indeed, 9P basis is the transaction : ie a Request followed
> by a Reply.
> So I think you should not pipeline the requests.
> Can a 9P specialist can confirm this ? If so ... I now understand why
> Fids are
> (or must be) choosen by the client.
>
>     Phil;
> > Well,  I've not read the protocol details for a while.
> > But from memory I thought it allowed this type of operation:
> >
> >     send: open,fid,file
> >     send: read,fid,args
> >     send: read,fid,args
> >
> >     (wait one rtt)
> >
> >     recv: open success/fail
> >     recv: read result / read error due to unknown fid
> >     recv: read result / read error due to unknown fid
> >
> > DF
> >
> >
>
>


--
- Passage Matthew 5:37:
   But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil.

[-- Attachment #2: Type: text/html, Size: 1710 bytes --]

Re: [9fans] About 9P ...

[Skip Tavakkolian]

> So I think you should not pipeline the requests.
> Can a 9P specialist can confirm this ?

not true.

Re: [9fans] About 9P ...

[Philippe Anel]

David Leimbach a écrit :
> Transactions are tagged right?  So you can, in fact, have many in 
> flight at once.
>
> Perhaps I'm missing what you meant by pipelined.
I meant sending a couple of requests on one fid before receiving
replies.

Yes transaction are tagged, this doesn't mean that requests are
serialized (processed in order).

See the example sent by Derek :

1   send: open,fid,file
2   send: read,fid,args
3   send: read,fid,args

    (wait one rtt)

4   recv: open success/fail
5   recv: read result / read error due to unknown fid
7   recv: read result / read error due to unknown fid

I see no reason for the server to not reply the 2nd request
before the 1st one : by example the server can receive all
the requests and distpach those reqs to different tasks.

Because all transactions are tagged ... this wouldn't break
9P. However, this doesn't work as expected.

That's why I think 9P was not designed to allow this. But
maybe I'm wrong.


Phil;

Re: [9fans] About 9P ...

[Philippe Anel]

My bad, I'm not a good english writter ... I meant replies to request are
not supposed to be returned in request order.

Skip Tavakkolian a écrit :
>> So I think you should not pipeline the requests.
>> Can a 9P specialist can confirm this ?
>>     
>
> not true.
>
>
>
>   

Re: [9fans] About 9P ...

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2310 bytes --]

On 6/22/07, Philippe Anel <xigh@free.fr> wrote:
>
> David Leimbach a écrit :
> > Transactions are tagged right?  So you can, in fact, have many in
> > flight at once.
> >
> > Perhaps I'm missing what you meant by pipelined.
> I meant sending a couple of requests on one fid before receiving
> replies.



That I believe is correct, there's no guarantee that a server has to process
your requests on a single fid in the order the requests are sent/received.

It will eventually reply to your requests in some order, and you'll be able
to tell what order they occurred since I believe 9p requires a transport
guaranteeing the sequence sent will be the sequence received.  (I'm a 9p
amateur though)


Yes transaction are tagged, this doesn't mean that requests are
> serialized (processed in order).
>
> See the example sent by Derek :
>
> 1   send: open,fid,file
> 2   send: read,fid,args
> 3   send: read,fid,args
>
>     (wait one rtt)
>
> 4   recv: open success/fail
> 5   recv: read result / read error due to unknown fid
> 7   recv: read result / read error due to unknown fid
>
> I see no reason for the server to not reply the 2nd request
> before the 1st one : by example the server can receive all
> the requests and distpach those reqs to different tasks.


Well you may have a point, but why not

1. send: open, fid, file
2. wait for reply to 1.
3. send read, fid, args
4. send read, fid, args
5. wait for either...
6. wait for remaining?

At that point you're still pipelining, and since you're reading presumably
into separate buffers, or different locations in the same buffer, who cares
about the order?



> Because all transactions are tagged ... this wouldn't break
> 9P. However, this doesn't work as expected.



Depends on what's expected :-)


That's why I think 9P was not designed to allow this. But
> maybe I'm wrong.



It's probably designed to allow what I just said, you can sequence some
operations, but then things that don't need to be sequenced could be
pipelined.

Different servers may behave differently, 9P makes no guarantee AFAIK.

Dave


Phil;
>
>
>


-- 
- Passage Matthew 5:37:
   But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil.

[-- Attachment #2: Type: text/html, Size: 3993 bytes --]

Re: [9fans] About 9P ...

[Philippe Anel]

Ok I agree ... as long as you don't expect the replies to be returned
in the same order as the requests, requests can be pipelined.

Therefore, now it makes sense to have FIDs chosen by the client.

Thank you.

>
> Well you may have a point, but why not
>
> 1. send: open, fid, file
> 2. wait for reply to 1.
> 3. send read, fid, args
> 4. send read, fid, args
> 5. wait for either...
> 6. wait for remaining?
>
> At that point you're still pipelining, and since you're reading
> presumably into separate buffers, or different locations in the same
> buffer, who cares about the order?
>
>
>
>     Because all transactions are tagged ... this wouldn't break
>     9P. However, this doesn't work as expected.
>
>
>
> Depends on what's expected :-)
>
>
>     That's why I think 9P was not designed to allow this. But
>     maybe I'm wrong.
>
>
>
> It's probably designed to allow what I just said, you can sequence
> some operations, but then things that don't need to be sequenced could
> be pipelined.
>
> Different servers may behave differently, 9P makes no guarantee AFAIK.
>
> Dave
>
>
>     Phil;
>
>
>
>
>
> --
> - Passage Matthew 5:37:
>    But let your communication be, Yea, yea; Nay, nay: for whatsoever
> is more than these cometh of evil.

Re: [9fans] About 9P ...

[Philippe Anel]

Sometimes, I whish I could think faster.

The fact request can be pipelined like this -I mean, by taking care of the
fact replies are not necessarly returned in requests order- doesn't not
mean that clients have to choose the FIDs. This would also work if FIDs
are choosen by the server (and even better because the client would not
have to take care of the order).

So unless Russ is right and that the fact FIDS are choosen by the client has
been decided arbitrarily ... I still have no answer to my question.

Phil;

Philippe Anel a écrit :
> Ok I agree ... as long as you don't expect the replies to be returned
> in the same order as the requests, requests can be pipelined.
>
> Therefore, now it makes sense to have FIDs chosen by the client.
>
> Thank you.
>
>>
>> Well you may have a point, but why not
>>
>> 1. send: open, fid, file
>> 2. wait for reply to 1.
>> 3. send read, fid, args
>> 4. send read, fid, args
>> 5. wait for either...
>> 6. wait for remaining?
>>
>> At that point you're still pipelining, and since you're reading 
>> presumably into separate buffers, or different locations in the same 
>> buffer, who cares about the order?
>>
>>  
>>
>>     Because all transactions are tagged ... this wouldn't break
>>     9P. However, this doesn't work as expected.
>>
>>
>> Depends on what's expected :-)
>>  
>>
>>     That's why I think 9P was not designed to allow this. But
>>     maybe I'm wrong.
>>
>>
>>
>> It's probably designed to allow what I just said, you can sequence 
>> some operations, but then things that don't need to be sequenced 
>> could be pipelined.
>>
>> Different servers may behave differently, 9P makes no guarantee AFAIK.
>>
>> Dave
>>  
>>
>>     Phil;
>>
>>
>>
>>
>>
>> -- 
>> - Passage Matthew 5:37:
>>    But let your communication be, Yea, yea; Nay, nay: for whatsoever 
>> is more than these cometh of evil. 
>
>
>

Re: [9fans] About 9P ...

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2573 bytes --]

On 6/22/07, Philippe Anel <xigh@free.fr> wrote:
>
>
> Sometimes, I whish I could think faster.
>
> The fact request can be pipelined like this -I mean, by taking care of the
> fact replies are not necessarly returned in requests order- doesn't not
> mean that clients have to choose the FIDs. This would also work if FIDs
> are choosen by the server (and even better because the client would not
> have to take care of the order).
>
> So unless Russ is right and that the fact FIDS are choosen by the client
> has
> been decided arbitrarily ... I still have no answer to my question.



I think servers often serve multiple clients, and therefore it's probably
best to leave some things client side.

Perhaps that's why google's not doing full SOAP anymore and just returning
XML that you have to parse yourself on the client side.

It may not really be arbitrary after all :-)


Phil;
>
> Philippe Anel a écrit :
> > Ok I agree ... as long as you don't expect the replies to be returned
> > in the same order as the requests, requests can be pipelined.
> >
> > Therefore, now it makes sense to have FIDs chosen by the client.
> >
> > Thank you.
> >
> >>
> >> Well you may have a point, but why not
> >>
> >> 1. send: open, fid, file
> >> 2. wait for reply to 1.
> >> 3. send read, fid, args
> >> 4. send read, fid, args
> >> 5. wait for either...
> >> 6. wait for remaining?
> >>
> >> At that point you're still pipelining, and since you're reading
> >> presumably into separate buffers, or different locations in the same
> >> buffer, who cares about the order?
> >>
> >>
> >>
> >>     Because all transactions are tagged ... this wouldn't break
> >>     9P. However, this doesn't work as expected.
> >>
> >>
> >> Depends on what's expected :-)
> >>
> >>
> >>     That's why I think 9P was not designed to allow this. But
> >>     maybe I'm wrong.
> >>
> >>
> >>
> >> It's probably designed to allow what I just said, you can sequence
> >> some operations, but then things that don't need to be sequenced
> >> could be pipelined.
> >>
> >> Different servers may behave differently, 9P makes no guarantee AFAIK.
> >>
> >> Dave
> >>
> >>
> >>     Phil;
> >>
> >>
> >>
> >>
> >>
> >> --
> >> - Passage Matthew 5:37:
> >>    But let your communication be, Yea, yea; Nay, nay: for whatsoever
> >> is more than these cometh of evil.
> >
> >
> >
>
>


-- 
- Passage Matthew 5:37:
   But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil.

[-- Attachment #2: Type: text/html, Size: 3710 bytes --]

Re: [9fans] About 9P ...

[Anant Narayanan]

David Leimbach wrote:
<snip>
> Perhaps that's why google's not doing full SOAP anymore and just
> returning XML that you have to parse yourself on the client side.

If they wanted to "leave" things to the client side, they wouldn't use
XML at all.

--
Anant

Re: [9fans] About 9P ...

[Skip Tavakkolian]

there were discussions about aysnc syscalls.  /sys/src/cmd/fcp.c is a
good example of why they're not needed.  streaming and long delay
networks can be handled this way too, as was pointed out (by Russ i
think) at iwp9.

Re: [9fans] About 9P ...

[Francisco J Ballesteros]

Skip wrote:
:there were discussions about aysnc syscalls.  /sys/src/cmd/fcp.c is a
:good example of why they're not needed.  streaming and long delay
:networks can be handled this way too, as was pointed out (by Russ i
:think) at iwp9.

But there is one problem. Consider "lc".

Usually you see
walk
clunk
walk
clunk
walk
open
write
clunk

and also
walk
stat clunk
walk
open
read
clunk

The problem is, how to know which RPCs to pack? I mean, to send without waiting
for reply. Note that several different fids might be used in these
dialogs, even though they
come in a small time window from the same process.
I tried to imagine how to bundle them, for using 9P over high latency links,
and couldn't find a good way of doing it (after iwp9).

In the end, I had to implement a couple of processes to bridge islands with good
internal latency, connected by poor links. They try to refrain from
sending requests by
using cached directory entries (and data) to try to make such runs in
one or two RPCs.
The hard part is making it work well both for regular files and for
synthesized files.
We have a version that works reasonable well for both, (only
connections multiplexed
over the same file, eg. clone opens, are not handled properly).

Our current version of the protocol has "file descriptors" that are fids in that
they "mean" a path for a file, and are not fids in that more than one
fid may map to
the same descriptor, and also in that paths are sent along with the
descriptor to make
the server stateless (or to make clients think so).

9P is just great for use when latency is reasonable (or not too bad,
with cfs), but to go further
away and still be comfortable using remote files, I'd say we need
another protocol.
I'd love to be proved wrong :)

Re: [9fans] About 9P ...

[Francisco J Ballesteros]

I don't think so. Because we try to use results from past, and not to
look ahead.


On 6/23/07, Roman Shaposhnick <rvs@sun.com> wrote:
> On Sat, 2007-06-23 at 01:48 +0200, Francisco J Ballesteros wrote:
> > Skip wrote:
> > :there were discussions about aysnc syscalls.  /sys/src/cmd/fcp.c is a
> > :good example of why they're not needed.  streaming and long delay
> > :networks can be handled this way too, as was pointed out (by Russ i
> > :think) at iwp9.
> >
> > But there is one problem. Consider "lc".
> >
> > Usually you see
> > walk
> > clunk
> > walk
> > clunk
> > walk
> > open
> > write
> > clunk
> >
> > and also
> > walk
> > stat clunk
> > walk
> > open
> > read
> > clunk
> >
> > The problem is, how to know which RPCs to pack?
>
>   Isn't it the very same problem that compilers have with
> instruction pipelining?
>    http://en.wikipedia.org/wiki/Instruction_pipelining
>
>   With an additional complication that you can't actually
> look ahead very often?
>
> Thanks,
> Roman.
>
>

Re: [9fans] About 9P ...

[Roman Shaposhnick]

On Sat, 2007-06-23 at 01:48 +0200, Francisco J Ballesteros wrote:
> Skip wrote:
> :there were discussions about aysnc syscalls.  /sys/src/cmd/fcp.c is a
> :good example of why they're not needed.  streaming and long delay
> :networks can be handled this way too, as was pointed out (by Russ i
> :think) at iwp9.
>
> But there is one problem. Consider "lc".
>
> Usually you see
> walk
> clunk
> walk
> clunk
> walk
> open
> write
> clunk
>
> and also
> walk
> stat clunk
> walk
> open
> read
> clunk
>
> The problem is, how to know which RPCs to pack?

  Isn't it the very same problem that compilers have with
instruction pipelining?
   http://en.wikipedia.org/wiki/Instruction_pipelining

  With an additional complication that you can't actually
look ahead very often?

Thanks,
Roman.

Re: [9fans] About 9P ...

[Russ Cox]

> So unless Russ is right and that the
> fact FIDS are choosen by the client has
> been decided arbitrarily ... I still have
> no answer to my question.

In the current version of 9P, there is no reason
why the client has to choose the fids (file ids).
You don't have to believe me, but it's true.

The client must choose the tags (request ids)
since it sends the first request message without
any input from the server.  The tags are what
enable having multiple messages outstanding.

There is an historical reason why fids are
chosen by the client, but like most historical
reasons, it no longer applies.  In very early
versions of 9P (before the first edition release),
there were no tags.  Replies were matched
based on fid and type, so fids had to be chosen
by the client, just as tags do now.

Tags were introduced on November 21, 1990:
http://swtch.com/go/fid2tag

Russ

Re: [9fans] About 9P ...

[Skip Tavakkolian]

> Tags were introduced on November 21, 1990:
> http://swtch.com/go/fid2tag

good question and great historical perspective.

is software archaeology a formal discipline yet?

Re: [9fans] About 9P ...

[Philippe Anel]

Thank you Russ.

Russ Cox a écrit :
>> So unless Russ is right and that the 
>> fact FIDS are choosen by the client has
>> been decided arbitrarily ... I still have 
>> no answer to my question.
>>     
>
> In the current version of 9P, there is no reason
> why the client has to choose the fids (file ids).
> You don't have to believe me, but it's true.
>
> The client must choose the tags (request ids)
> since it sends the first request message without 
> any input from the server.  The tags are what 
> enable having multiple messages outstanding.
>
> There is an historical reason why fids are 
> chosen by the client, but like most historical
> reasons, it no longer applies.  In very early
> versions of 9P (before the first edition release),
> there were no tags.  Replies were matched
> based on fid and type, so fids had to be chosen
> by the client, just as tags do now. 
>
> Tags were introduced on November 21, 1990:
> http://swtch.com/go/fid2tag
>
> Russ
>
>
>
>   

Re: [9fans] About 9P ...

[Francisco J Ballesteros]

Don't know, but we have strong group on that on the other corridor of
the building.
They count lines, and count, and count...
They use Linux mostly, and have a huge ammount of lines to count.

On 6/23/07, Skip Tavakkolian <9nut@9netics.com> wrote:
> > Tags were introduced on November 21, 1990:
> > http://swtch.com/go/fid2tag
>
> good question and great historical perspective.
>
> is software archaeology a formal discipline yet?
>
>

Re: [9fans] About 9P ...

[erik quanstrom]

> 9P is just great for use when latency is reasonable (or not too bad,
> with cfs), but to go further
> away and still be comfortable using remote files, I'd say we need
> another protocol.
> I'd love to be proved wrong :)

are there any protocols that deal well with latency?

the only way i know to deal with latency is to do some
sort of tagged queueing.  (perhaps i don't know enough
computer history.)  unfortunately, this doesn't
work if part of the data you need depend on some
prior part; a conversation means ping-pong communication.

the great news for 9p is that latency is decreasing.  in 
/sys/doc/net/net.ps, the IL/ether latency is listed as
having a latency of 1420µs.  my home fileserver
has a 57µs latency to my cpu server. (the fileserver is a
pIII with a $19 rtl8169.)

even custom Cyclone interface between the file server and 
the cpu server is listed as having a latency of 375µs.

let's hope the same holds true for wireless networking.

- erik

Re: [9fans] About 9P ...

[Francisco J Ballesteros]

Yes, but latency from Madrid to Greece is 150ms (rtt, icmp) and that
I'd say is going
to stay that way for a long time. To the US (east coast) it's 120ms.

I still like to use my system while in the move.

Regarding protocols, lbfs is nice, but focuses on bandwidth and so, it
requires multiple
RPCs for things that, for example, NFSv4, does in one or two.

On 6/23/07, erik quanstrom <quanstro@coraid.com> wrote:
> > 9P is just great for use when latency is reasonable (or not too bad,
> > with cfs), but to go further
> > away and still be comfortable using remote files, I'd say we need
> > another protocol.
> > I'd love to be proved wrong :)
>
> are there any protocols that deal well with latency?
>
> the only way i know to deal with latency is to do some
> sort of tagged queueing.  (perhaps i don't know enough
> computer history.)  unfortunately, this doesn't
> work if part of the data you need depend on some
> prior part; a conversation means ping-pong communication.
>
> the great news for 9p is that latency is decreasing.  in
> /sys/doc/net/net.ps, the IL/ether latency is listed as
> having a latency of 1420µs.  my home fileserver
> has a 57µs latency to my cpu server. (the fileserver is a
> pIII with a $19 rtl8169.)
>
> even custom Cyclone interface between the file server and
> the cpu server is listed as having a latency of 375µs.
>
> let's hope the same holds true for wireless networking.
>
> - erik
>

Re: [9fans] About 9P ...

[Lucio De Re]

On Sat, 2007-06-23 at 08:08 -0400, erik quanstrom wrote:
> are there any protocols that deal well with latency?
>
IP does by design.  The correct approach to the problem is the
sliding-window technique and I can only presume that applications that
suffer from high latency are written to conflict with the underlying
protocols.

I mean, we've had X-modem since the ark.  I've tried to avoid becoming
too familiar with 9P to be able to comment because my mind isn't as
sharp as it was when I produced end-to-end protocols to run on 300 bps
lines, but maybe somebody can explain to me whether 9P is one such
ill-mannered application.  From the immediately preceding discussion, it
doesn't seem so, but I do not speak from a position of knowledge.

++L

Re: [9fans] About 9P ...

[Uriel]

I thought russ (and charles and sape?) proposed a good solutions to
the latency issues in 9P at IWP9. Actually as far as I can remember it
was the only change to 9P that was accepted as justified.

My memory is bad, probably got the details wrong but I think there
were two approaches: one by allowing multiple requests with the same
tag to be grouped, and another with an extra flag filed that indicates
if more messages in a group are on their way.

I still don't understand what could be wrong with either approach (but
again, I don't even remember the details of either) and why nemo had
to invent yet another protocol. Maybe someone can enlighten my usual
ignorance.

uriel

On 6/23/07, erik quanstrom <quanstro@coraid.com> wrote:
> > 9P is just great for use when latency is reasonable (or not too bad,
> > with cfs), but to go further
> > away and still be comfortable using remote files, I'd say we need
> > another protocol.
> > I'd love to be proved wrong :)
>
> are there any protocols that deal well with latency?
>
> the only way i know to deal with latency is to do some
> sort of tagged queueing.  (perhaps i don't know enough
> computer history.)  unfortunately, this doesn't
> work if part of the data you need depend on some
> prior part; a conversation means ping-pong communication.
>
> the great news for 9p is that latency is decreasing.  in
> /sys/doc/net/net.ps, the IL/ether latency is listed as
> having a latency of 1420µs.  my home fileserver
> has a 57µs latency to my cpu server. (the fileserver is a
> pIII with a $19 rtl8169.)
>
> even custom Cyclone interface between the file server and
> the cpu server is listed as having a latency of 375µs.
>
> let's hope the same holds true for wireless networking.
>
> - erik
>

Re: [9fans] About 9P ...

[Francisco J Ballesteros]

It would be fine to just use the same tag as suggested by Russ ans Sape
if you just happen to have several 9P requests that you know that have
to be issued
to serve a particular call. For example, say you add a library
function (or syscall)
get(),
you could send Twalk tag3 nfid F Topen fid F, Tread fid F, Tread fid F, ....
all with the same tag, without waiting for replies, and the server,
knowning they have
the same tag, would sever Topen, Tread, ... only if Twalk worked. That
would batch
all the requests, as intended.

Now, say you have a program that does:
if (stat(afile)){
   open(afile)
   read(afile)
   close(afile)
}

By the time of the stat you could have read the initial part of the
file along with its
metadata (maybe the entire file!, if it's small). Also, some other
program might be using
the same file, and you'd send the RPCs on its behalf.

If instead of changing the protocol, you use intermediate processes
that do this "caching",
this is what happens:

by the time of the stat, the process does a get (even if it has a
cache entry for the file, to
check for cache validity). That get, retrieves initial data for the file.

Because the entry is known valid, any call to read bytes (and any stat
done by any other
process in the mean time) would be served from the local process, no
further RPCs.

In particular, the open() above would use the entry in the cache and
would not walk to the file again. This does not hurt (much) coherency.
Suppose the file was removed after open,
you cannot tell if what happen is that it was removed after read
(going to the server to do the
read) or what happen is that it was removed after open (and data was
served from the the
data just read during open, or during stat).

As this caching thing changes the semantics quite a bit, it seemed ok
to use the simplest
protocol that we could imagine that would implement such behaviour,
instead of batching
9p requests just to have 9p on the wire.

Note that having two programs <file calls> <-> op  and op <-> styx
means that nobody
but these two programs has to be aware of the new protocol. It's
buried deep in the
bad latency communications link, and everyone else thinks that styx is
the protocol spoken
everywhere.

Again, if anyone sees anything wrong if this approach, even though
it's already implemented
and seems to be working fine, I'd love to find out a better way.



On 6/23/07, Uriel <uriel99@gmail.com> wrote:
> I thought russ (and charles and sape?) proposed a good solutions to
> the latency issues in 9P at IWP9. Actually as far as I can remember it
> was the only change to 9P that was accepted as justified.
>
> My memory is bad, probably got the details wrong but I think there
> were two approaches: one by allowing multiple requests with the same
> tag to be grouped, and another with an extra flag filed that indicates
> if more messages in a group are on their way.
>
> I still don't understand what could be wrong with either approach (but
> again, I don't even remember the details of either) and why nemo had
> to invent yet another protocol. Maybe someone can enlighten my usual
> ignorance.
>
> uriel
>
> On 6/23/07, erik quanstrom <quanstro@coraid.com> wrote:
> > > 9P is just great for use when latency is reasonable (or not too bad,
> > > with cfs), but to go further
> > > away and still be comfortable using remote files, I'd say we need
> > > another protocol.
> > > I'd love to be proved wrong :)
> >
> > are there any protocols that deal well with latency?
> >
> > the only way i know to deal with latency is to do some
> > sort of tagged queueing.  (perhaps i don't know enough
> > computer history.)  unfortunately, this doesn't
> > work if part of the data you need depend on some
> > prior part; a conversation means ping-pong communication.
> >
> > the great news for 9p is that latency is decreasing.  in
> > /sys/doc/net/net.ps, the IL/ether latency is listed as
> > having a latency of 1420µs.  my home fileserver
> > has a 57µs latency to my cpu server. (the fileserver is a
> > pIII with a $19 rtl8169.)
> >
> > even custom Cyclone interface between the file server and
> > the cpu server is listed as having a latency of 375µs.
> >
> > let's hope the same holds true for wireless networking.
> >
> > - erik
> >
>

Re: [9fans] About 9P ...

[Lucio De Re]

On Sat, 2007-06-23 at 15:36 +0200, Francisco J Ballesteros wrote:

>
> Now, say you have a program that does:
> if (stat(afile)){
>    open(afile)
>    read(afile)
>    close(afile)
> }
>
So those are 9P's bad manners.  How many such idioms can one subsume
into a "caching" process?  Because in my opinion it makes more sense, as
I understand to have been Russ and Sape's proposal, to provide a
sliding-window version of the above that sends all of them at once:

	if (stat(afile), open(afile), read(afile), close(afile)) {
          ...
        } else {
          perror ("");
        }

Somehow, we may need to differentiate the error return and it sure looks
like programmers need to be aware of this implementation technique, but
is it preferable to construct a specialised cache that knows to generate
a get() whenever a stat() is performed?  The latter doesn't really scale
to other potential scenarios and may add unwanted overheads.

++L

Re: [9fans] About 9P ...

[Francisco J Ballesteros]

Note that

if (stat()){
   open
   read
   close
}

is not new code. It's code in ls, 8c, mk, acme, etc. etc. etc.
The point is how to make that code work nicely across bad latency links.
But you do not want to change that code.

Also, there's no overhead if you don't use our stuff while "within the
building".

On 6/23/07, Lucio De Re <lucio@proxima.alt.za> wrote:
> On Sat, 2007-06-23 at 15:36 +0200, Francisco J Ballesteros wrote:
>
> >
> > Now, say you have a program that does:
> > if (stat(afile)){
> >    open(afile)
> >    read(afile)
> >    close(afile)
> > }
> >
> So those are 9P's bad manners.  How many such idioms can one subsume
> into a "caching" process?  Because in my opinion it makes more sense, as
> I understand to have been Russ and Sape's proposal, to provide a
> sliding-window version of the above that sends all of them at once:
>
>         if (stat(afile), open(afile), read(afile), close(afile)) {
>           ...
>         } else {
>           perror ("");
>         }
>
> Somehow, we may need to differentiate the error return and it sure looks
> like programmers need to be aware of this implementation technique, but
> is it preferable to construct a specialised cache that knows to generate
> a get() whenever a stat() is performed?  The latter doesn't really scale
> to other potential scenarios and may add unwanted overheads.
>
> ++L
>
>
>

Re: [9fans] About 9P ...

[Rob Pike]

>
> Tags were introduced on November 21, 1990:
> http://swtch.com/go/fid2tag

Thanksgiving weekend.  We kicked everyone off for the weekend
and redid 9P to fix all the problems we'd figured out so far.

-rob

Re: [9fans] About 9P ...

[Martin Atkins]

Francisco J Ballesteros wrote:
> Don't know, but we have strong group on that on the other corridor of
> the building.
> They count lines, and count, and count...
> They use Linux mostly, and have a huge ammount of lines to count.
Of course :-)

But then not many people are "fortunate enough" to see the lines in
other systems, in order to be able to count them....

[9fans] 9P optimization

[Lyndon Nerenberg]

Before we think about turning 9P into NFSv4, it would be useful to examine
the current use of cfs, and collect some stats on how effective it is.

Re: [9fans] 9P optimization

[Francisco J Ballesteros]

The problem with cfs is that you still suffer the RPC to check for validity.
With 100ms of RTT, count 5 and you have a second. Also, for synthesized files,
the cache does not save you from doing a walk, open, read.

That sayd, I don't think a change in 9p is needed. Translating to
another protocol for low links
suffices.

On 6/24/07, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
> Before we think about turning 9P into NFSv4, it would be useful to examine
> the current use of cfs, and collect some stats on how effective it is.
>

Re: [9fans] 9P optimization

[Lyndon Nerenberg]

> The problem with cfs is that you still suffer the RPC to check for validity.
> With 100ms of RTT, count 5 and you have a second. Also, for synthesized
> files,
> the cache does not save you from doing a walk, open, read.

I'll be accused of being a heretic, but let me claim that much of the
design of IMAP is geared towards avoiding those RTTs for that very reason.
A good IMAP client wins by doing intelligent cacheing, and it gets there
by using the subtle aspects of the protocol that accomodate that.

Much as people slag the protocol, it does work well over slow (high
latency) links, given a client that understands the protocol (sadly, most
don't).

I'm very curious about the efficiency of cfs, but my P9 environment
doesn't have a remote terminal.  It would be interesting to instrument cfs
and collect some stats on the protocol behaviour behind the cache ...

--lyndon

Re: [9fans] 9P optimization

[ron minnich]

I wrote (a long time ago) a caching file system, for root file
systems, described here:
http://mbgokhale.org/rminnich/job/autocacher/autocache.ps, that did
the simple thing: once you have the file, don't bother checking with
the server until time + DeltaT, This might not work well for you, but
in 1989, it worked wonderfully well as a root file system because:
1. most files in most root file systems are never even looked at
(particularly man pages!)
2. They don't change much anyway -- think about it -- how often do
people like to upgrade :-)

So it is not always the case that you need to stat the cached files on
each access. In fact, we watched NFS traffic on our backbone at the
SRC for several months, and, as many other people have found, most
files are mostly read.

This type of optimization is fairly trivial for CFS, and would end up
covering most files (see above); the few read-write ones left would
not really hurt you too much. You could do much better in 9p because
it is stateful.

I have a new implementation of the autocacher (it's only for NFS)
which I wrote ca. 1999, before I got to LANL; I had lost it but found
it on my last day of work at LANL. I'll try to get it on my web page.
It's a very simple program.

thanks

ron

Re: [9fans] 9P optimization

[Bruce Ellis]

rangbooming has lead to extreme optimiazations.  i'll let skip
comment on it (he's asleep i guess).  but i couldn't have much
worse RTT (sydney-seattle), and the traffic seems a lot lower now,
and the latency has improved extraordinarily.

brucee

On 6/24/07, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:
> > The problem with cfs is that you still suffer the RPC to check for validity.
> > With 100ms of RTT, count 5 and you have a second. Also, for synthesized
> > files,
> > the cache does not save you from doing a walk, open, read.
>
> I'll be accused of being a heretic, but let me claim that much of the
> design of IMAP is geared towards avoiding those RTTs for that very reason.
> A good IMAP client wins by doing intelligent cacheing, and it gets there
> by using the subtle aspects of the protocol that accomodate that.
>
> Much as people slag the protocol, it does work well over slow (high
> latency) links, given a client that understands the protocol (sadly, most
> don't).
>
> I'm very curious about the efficiency of cfs, but my P9 environment
> doesn't have a remote terminal.  It would be interesting to instrument cfs
> and collect some stats on the protocol behaviour behind the cache ...
>
> --lyndon
>

Re: [9fans] 9P optimization

[Steve Simon]

> http://mbgokhale.org/rminnich/job/autocacher/autocache.ps

Interesting idea, I must read up on what else has been published on cacheing
to avoid high latency links.

It would be fun to modify cfs to have a couple of cache coherency algorithms,
the timeout you describe could be one, Boyds ftp cache rule is also interesting -
the timeout value is proportional to how long ago the file was last written, so
old files tend to be stable.

The cache could be dynamic on the device type on which the file lives,
fossil/kfs/kenfs get timeouts, others are assumed to be synthetic files and so
should have their verson numbers checked every time as cfs does now.

We could even implement a /dev/fossilchg file which contains the dirstat(2) info
of all modified files as they change. This would allow persistant (or at least
long lived) connections to attempt to keep a near consistant view of the file.
That is any file which has had an update since the local cfs has been started
would have a known state (be it consistant or inconsisitant); similarly any file
who had to have its version checked would have a known state also.

This is of course a seperate issue to the coelessing of multiple p9 operations
in a single packet as, though as previous posters pointed out it would help cfs too.

my 2¢

-Steve

Re: [9fans] 9P optimization

[Lyndon Nerenberg]

> It would be fun to modify cfs to have a couple of cache coherency algorithms,
> the timeout you describe could be one, Boyds ftp cache rule is also interesting -
> the timeout value is proportional to how long ago the file was last written, so
> old files tend to be stable.

I loath the idea of pushing state into 9P, but I wonder if a combination
of client side cfs with a kqueue-like 'file is modified' server event
might not solve most of the problem.  Layered into aan perhaps?


--lyndon

   What is this talk of 'release'?  Klingons do not make software
   'releases'.  Our software 'escapes' leaving a bloody trail of designers and
   quality assurance people in its wake.

Re: [9fans] 9P optimization

[Steve Simon]

> I loath the idea of pushing state into 9P, but I wonder if a combination
> of client side cfs with a kqueue-like 'file is modified' server event
> might not solve most of the problem.  Layered into aan perhaps?

Not sure if I missunderstand you but I am _not_ suggesting pushing any
state into p9.

My thoughts where modifying cfs heuristics to reduce the probability of
waiting for a coherence check (which could be quite a few RTTs).

In the limit I also suggested providing a synthetic file which contains
dirstat(1) info on modified files as they change - which I think is what
you mean by a "kqueue-like 'file is modified' server".

The problem with implementing this as a server program on the fileserver machine
is that _all_ fossil/kfs/kenfs file accesses would have to pass through this server
to be sure that a when a file is changed by somone else you are informed.
As fossil (what I use)  already knows this info I thougt it would be as well for it
to provide this file itself.

hope this clarifies.

-Steve

Re: [9fans] 9P optimization

[erik quanstrom]

rather than go to these extremes, why not use a local root?
run replica in a cron job if necessary.  a paired-down root
can fit in 32MB more-or-less.

i use import so i can run acme locally, and cpu in acme windows to
access machines at work.  this way only command stdin, stdout, stderr
and the files i edit cross the wire.

- erik

Re: [9fans] 9P optimization

[Lyndon Nerenberg]

> Not sure if I missunderstand you but I am _not_ suggesting pushing any
> state into p9.

> My thoughts where modifying cfs heuristics to reduce the probability of
> waiting for a coherence check (which could be quite a few RTTs).

We're on the same track.  This stuff would fit in cfs just as easily.
What I'm curious about is if a 'the file was modified' callback would help
-- basically, just a notification to flush the cache, which would let the
client be more aggressive about holding blocks in the cache without having
to ping the server on every read.

9P wouldn't need any mods -- this would work easily as a filter pushed on
the 9fs connection.  But it would need a callback facility to notify the
server side of the cache filter that the file had changed.

--lyndon

Re: [9fans] 9P optimization

[Lyndon Nerenberg]

> rather than go to these extremes, why not use a local root?

Even with a local root, importing stuff over (say) wifi can be painful.

--lyndon

   NT as a file server is faster than a dead bat carrying Post-It notes
   underwater. But not by much.

Re: [9fans] 9P optimization

[Skip Tavakkolian]

> rangbooming has lead to extreme optimiazations.  i'll let skip
> comment on it (he's asleep i guess).  but i couldn't have much
> worse RTT (sydney-seattle), and the traffic seems a lot lower now,
> and the latency has improved extraordinarily.

behavior of windows exploder (rer) and network latency really magnify
this problem.  we cache the walks, assisted by staling and usage
tracking.  staling >1×rtt will satiate incessant walk requests from
cache.

it is useful for what we're doing.  brucee has an idea on how it could
be applied to something like exportfs, for general use.

regarding locks or cache expiry notification that others suggested,
the proposal in snspaper was to address that, but in the end we all
agreed that if you need it, a control file between the fileserver and
the importing side would be sufficient.

Re: [9fans] 9P optimization

[erik quanstrom]

> Even with a local root, importing stuff over (say) wifi can be painful.

perhaps there's something about your setup that's different than
mine.  even mounting root over a wireless connection is not a problem.
my desktop machine is connected a shared (with a dozen other computers)
wireless bridge.  the ping time of 3-4ms is not noticable unless one starts
compiling the kernel locally.  but cpu solves that problem quite nicely.

datakit terminals had similar latency and worse bandwidth.

at home, i have a 40ms ping time (surely your wireless connection is
faster) to work servers.  i sometimes notice just a slight delay saving
files over an encrypted, imported connection.  but since i compile over
a cpu session, it's just not an issue.

- erik

Re: [9fans] 9P optimization

[ron minnich]

On 6/24/07, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:

> 9P wouldn't need any mods -- this would work easily as a filter pushed on
> the 9fs connection.  But it would need a callback facility to notify the
> server side of the cache filter that the file had changed.

callback? Couldn't you just post a read on a file in the server and
have a thread hang on that?

ron

Re: [9fans] 9P optimization

[Dave Eckhardt]

There is some literature on the topic, e.g., AFS callbacks
(later called "leases" in the NFS world) and Coda's variable-
granularity callbacks... the problems aren't exactly the same,
but it seems like there might well be an opportunity to learn
from existing experience.

This describes experience gained from a system with actual
users (i.e., isn't just a simulation study):

  Exploiting Weak Connectivity for Mobile File Access
  http://www.cs.cmu.edu/~coda/docdir/s15.pdf

Dave Eckhardt

Re: [9fans] 9P optimization

[Lyndon Nerenberg]

> perhaps there's something about your setup that's different than
> mine.

There is.  When I tried it it was over a 11 Mb/s wireless link with much
worse latency than you have.  (We were seeing 200+ms RTTs to anywhere that
mattered, with 15-20% packet loss.)

--lyndon

   I love to sing, and I love to drink scotch. Most people would rather
   hear me drink scotch.
   			-- George Burns

Re: [9fans] 9P optimization

[Lyndon Nerenberg]

> callback? Couldn't you just post a read on a file in the server and
> have a thread hang on that?

I was thinking more along the lines of a notify server where you would
register pathnames of interest and post a read on a file that would
receive the event notifications.

--lyndon

   Never hit a man with glasses.  Hit him with a baseball bat.

Re: [9fans] 9P optimization

[erik quanstrom]

that's pretty massive packet loss.  tcp is likely to suffer mightily.

i used dialup until last year.  the rtt time to the gatway was 150ms and
the provider was proud enough of their network to push the default mtu
to 1500 bytes.  i never measured any packet loss.

perhaps you would have better luck with dialup?

- erik

On Mon Jun 25 13:48:55 EDT 2007, lyndon@orthanc.ca wrote:
> There is.  When I tried it it was over a 11 Mb/s wireless link with much
> worse latency than you have.  (We were seeing 200+ms RTTs to anywhere that
> mattered, with 15-20% packet loss.)
>
> --lyndon