[9fans] 9p vs http

[9fans] 9p vs http

[Sam Watkins]

hi,

I am wondering what you think about the capabilities of 9p compared to
http/1.1.  Perhaps this seems like an odd comparison, but I think 9p and http
are broadly similar in purpose and functionality.  While writing a simple
webserver, I got to thinking that http is really a very capable protocol.

http is text-based, it supports pipelining and arbitraty metadata.  As far as I
know, 9p does not support pipelining nor arbitraty metadata.  It seems to me
that these are big advantages for http.  9p supports walking; are there other
things 9p can do which http cannot, which give 9p a significant advantage?

Am I correct, that 9p does not support pipelining?  I suppose this would be a
big problem.  For example, with http pipelining one may ask a server to HEAD
(like stat) 10,000 files together, without having to wait for the responses.
Over a high latency link (e.g. Australia -> USA), this might save perhaps an
hour of waiting.

Such an asyncronous interface might be useful even when accessing local disks -
if the filesystem receives 100 open/read/stat requests bundled together, it
might optimise disk access to minimise seeking, as is commonly done for writes.

By the way, I read the other day on this list that there is no need to improve
cat(1).  Well for me, I still feel that the command `cat` without args should
concatenate 0 files (producing no output), not copy stdin to stdout!


Sam

Re: [9fans] 9p vs http

[John Floren]

Please see lsub's Op and my Streaming talk at the most recent IWP9.

Also, regarding 'cat', the behavior of many basic tools is that,
barring any file arguments, they take stdin as input and output to
stdout, so cat's behavior makes sense to me.

On Sun, Nov 14, 2010 at 10:25 PM, Sam Watkins <sam@nipl.net> wrote:
> hi,
>
> I am wondering what you think about the capabilities of 9p compared to
> http/1.1.  Perhaps this seems like an odd comparison, but I think 9p and http
> are broadly similar in purpose and functionality.  While writing a simple
> webserver, I got to thinking that http is really a very capable protocol.
>
> http is text-based, it supports pipelining and arbitraty metadata.  As far as I
> know, 9p does not support pipelining nor arbitraty metadata.  It seems to me
> that these are big advantages for http.  9p supports walking; are there other
> things 9p can do which http cannot, which give 9p a significant advantage?
>
> Am I correct, that 9p does not support pipelining?  I suppose this would be a
> big problem.  For example, with http pipelining one may ask a server to HEAD
> (like stat) 10,000 files together, without having to wait for the responses.
> Over a high latency link (e.g. Australia -> USA), this might save perhaps an
> hour of waiting.
>
> Such an asyncronous interface might be useful even when accessing local disks -
> if the filesystem receives 100 open/read/stat requests bundled together, it
> might optimise disk access to minimise seeking, as is commonly done for writes.
>
> By the way, I read the other day on this list that there is no need to improve
> cat(1).  Well for me, I still feel that the command `cat` without args should
> concatenate 0 files (producing no output), not copy stdin to stdout!
>
>
> Sam
>
>
>

Re: [9fans] 9p vs http

[Bruce Ellis]

i'm with john

On Mon, Nov 15, 2010 at 3:20 PM, John Floren <slawmaster@gmail.com> wrote:
> Please see lsub's Op and my Streaming talk at the most recent IWP9.
>
> Also, regarding 'cat', the behavior of many basic tools is that,
> barring any file arguments, they take stdin as input and output to
> stdout, so cat's behavior makes sense to me.
>
> On Sun, Nov 14, 2010 at 10:25 PM, Sam Watkins <sam@nipl.net> wrote:
>> hi,
>>
>> I am wondering what you think about the capabilities of 9p compared to
>> http/1.1.  Perhaps this seems like an odd comparison, but I think 9p and http
>> are broadly similar in purpose and functionality.  While writing a simple
>> webserver, I got to thinking that http is really a very capable protocol.
>>
>> http is text-based, it supports pipelining and arbitraty metadata.  As far as I
>> know, 9p does not support pipelining nor arbitraty metadata.  It seems to me
>> that these are big advantages for http.  9p supports walking; are there other
>> things 9p can do which http cannot, which give 9p a significant advantage?
>>
>> Am I correct, that 9p does not support pipelining?  I suppose this would be a
>> big problem.  For example, with http pipelining one may ask a server to HEAD
>> (like stat) 10,000 files together, without having to wait for the responses.
>> Over a high latency link (e.g. Australia -> USA), this might save perhaps an
>> hour of waiting.
>>
>> Such an asyncronous interface might be useful even when accessing local disks -
>> if the filesystem receives 100 open/read/stat requests bundled together, it
>> might optimise disk access to minimise seeking, as is commonly done for writes.
>>
>> By the way, I read the other day on this list that there is no need to improve
>> cat(1).  Well for me, I still feel that the command `cat` without args should
>> concatenate 0 files (producing no output), not copy stdin to stdout!
>>
>>
>> Sam
>>
>>
>>
>
>

Re: [9fans] 9p vs http

[Sam Watkins]

On Sun, Nov 14, 2010 at 11:20:00PM -0500, John Floren wrote:
> Please see lsub's Op and my Streaming talk at the most recent IWP9.

Ok, thanks.  I did not know that 9p has latency problems even when reading a
single file.  I was talking about pipelining, where you can ask the server to
send a dozen files or chunks all of metadata all in a single packet.  As I
said, I think this might be useful even within a site.

Do you think http has any disadvantages compared to 9p?
Could it be used instead of 9p in a similar role?


Example of http pipelining:

client sends:

	HEAD / HTTP/1.1
	Host: iwp9.org

	HEAD /slides/floren.pdf HTTP/1.1
	Host: iwp9.org


server replies:

	HTTP/1.1 200 OK
	Server: Plan9
	Date: Mon, 15 Nov 2010 05:06:10 GMT
	ETag: "2e3c1v24"
	Content-Length: 10382
	Last-Modified: Fri, 15 Oct 2010 14:14:24 GMT
	Content-Type: text/html

	HTTP/1.1 200 OK
	Server: Plan9
	Date: Mon, 15 Nov 2010 05:06:10 GMT
	ETag: "2e390v10"
	Content-Length: 122477
	Last-Modified: Wed, 13 Oct 2010 23:55:44 GMT
	Content-Type: application/pdf


> Also, regarding 'cat', the behavior of many basic tools is that,
> barring any file arguments, they take stdin as input and output to
> stdout, so cat's behavior makes sense to me.

Yes, I suggest the other tools are wrong too!  The '0 is special' behaviour may
be convenient for interactive use, but it's a disaster for shell scripting.
I already argued about this here at length and AFAIK did not convince anyone,
so there's not much point trying it again.


Sam

Re: [9fans] 9p vs http

[John Floren]

On Mon, Nov 15, 2010 at 12:16 AM, Sam Watkins <sam@nipl.net> wrote:
> On Sun, Nov 14, 2010 at 11:20:00PM -0500, John Floren wrote:
>> Please see lsub's Op and my Streaming talk at the most recent IWP9.
>
> Ok, thanks.  I did not know that 9p has latency problems even when reading a
> single file.  I was talking about pipelining, where you can ask the server to
> send a dozen files or chunks all of metadata all in a single packet.  As I
> said, I think this might be useful even within a site.
>
> Do you think http has any disadvantages compared to 9p?
> Could it be used instead of 9p in a similar role?
>
>
> Example of http pipelining:
>
> client sends:
>
>        HEAD / HTTP/1.1
>        Host: iwp9.org
>
>        HEAD /slides/floren.pdf HTTP/1.1
>        Host: iwp9.org
>
>
> server replies:
>
>        HTTP/1.1 200 OK
>        Server: Plan9
>        Date: Mon, 15 Nov 2010 05:06:10 GMT
>        ETag: "2e3c1v24"
>        Content-Length: 10382
>        Last-Modified: Fri, 15 Oct 2010 14:14:24 GMT
>        Content-Type: text/html
>
>        HTTP/1.1 200 OK
>        Server: Plan9
>        Date: Mon, 15 Nov 2010 05:06:10 GMT
>        ETag: "2e390v10"
>        Content-Length: 122477
>        Last-Modified: Wed, 13 Oct 2010 23:55:44 GMT
>        Content-Type: application/pdf
>
[snipped]
>
>
> Sam
>

You might be interested in Oleg Kiselyov's HTTPFS paper, which
basically put a filesystem frontend on HTTP, so he could give URLs as
arguments to programs and have them treated as normal files
(translating reads and writes into GET and PUT with caching).

I'm not sure how useful pipelining would be in the context of 9P. We
have a very tight relationship between each libc function (open, read,
write, close) and the 9P messages (Topen, Tread, Twrite, Tclunk), and
I just can't see where this pipelining thing would go in. If you want
to open and read two files, you call open() twice, get two different
fds, and read and write from those. Of course, when you're reading
from one file, nothing's happening with the other file; my streams
system (the paper and code should be out soon) would help alleviate
some of these problems.


John

Re: [9fans] 9p vs http

[Gorka Guardiola]

On Mon, Nov 15, 2010 at 6:16 AM, Sam Watkins <sam@nipl.net> wrote:
> On Sun, Nov 14, 2010 at 11:20:00PM -0500, John Floren wrote:
>> Please see lsub's Op and my Streaming talk at the most recent IWP9.
>
> Ok, thanks.  I did not know that 9p has latency problems even when reading a
> single file.  I was talking about pipelining, where you can ask the server to
> send a dozen files or chunks all of metadata all in a single packet.  As I
> said, I think this might be useful even within a site.
>
> Do you think http has any disadvantages compared to 9p?

Permissions, namespaces...

-- 
- curiosity sKilled the cat

Re: [9fans] 9p vs http

[Gorka Guardiola]

On Mon, Nov 15, 2010 at 3:09 PM, Gorka Guardiola <paurea@gmail.com> wrote:
> On Mon, Nov 15, 2010 at 6:16 AM, Sam Watkins <sam@nipl.net> wrote:
>> On Sun, Nov 14, 2010 at 11:20:00PM -0500, John Floren wrote:
>>> Please see lsub's Op and my Streaming talk at the most recent IWP9.
>>
>> Ok, thanks.  I did not know that 9p has latency problems even when reading a
>> single file.  I was talking about pipelining, where you can ask the server to
>> send a dozen files or chunks all of metadata all in a single packet.  As I
>> said, I think this might be useful even within a site.
>>
>> Do you think http has any disadvantages compared to 9p?
>
> Permissions, namespaces...

By namespaces I mean qid's , the notion that a file is the same if the
name isn't.


-- 
- curiosity sKilled the cat

Re: [9fans] 9p vs http

[roger peppe]

On 15 November 2010 14:15, Gorka Guardiola <paurea@gmail.com> wrote:
> By namespaces I mean qid's , the notion that a file is the same if the
> name isn't.

mind you, that's problematic in 9p. the qid can be the same even if the
file is different:

% ls -lqd /n/dump/2006/0707/usr/rog
(000000000003d540 1122 80) d-rwxr-xr-x M 42850 rog rog 0 Jun  7  2005
/n/dump/2005/0707/usr/rog
% ls -lqd /n/dump/2006/0707/usr/rog
(000000000003d540 1157 80) d-rwxr-xr-x M 42850 rog rog 0 Jun 12  2006
/n/dump/2006/0707/usr/rog

they have the same qid but they're different directories
with different contents.

it's difficult to do a good job when embedding file systems, as the qid space is
fixed. you can cheat, by rewriting the qid when the file is actually opened
to paper over the most obvious problems, but it's still not great.

perhaps things would be easier if qids were sufficiently large (e.g.
sha1 hash sized)
that an embedding file server could make up new qids (or create them by xoring
in other info) without fear of clashes.

re: 9p pipelining - i've been working on a little experiment related
to this. i'll be sharing the
results in a little while.

Re: [9fans] 9p vs http

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2218 bytes --]

On Sun, Nov 14, 2010 at 7:25 PM, Sam Watkins <sam@nipl.net> wrote:

> hi,
>
> I am wondering what you think about the capabilities of 9p compared to
> http/1.1.  Perhaps this seems like an odd comparison, but I think 9p and
> http
> are broadly similar in purpose and functionality.  While writing a simple
> webserver, I got to thinking that http is really a very capable protocol.
>
> http is text-based, it supports pipelining and arbitraty metadata.  As far
> as I
> know, 9p does not support pipelining nor arbitraty metadata.  It seems to
> me
> that these are big advantages for http.  9p supports walking; are there
> other
> things 9p can do which http cannot, which give 9p a significant advantage?
>
> Am I correct, that 9p does not support pipelining?  I suppose this would be
> a
> big problem.  For example, with http pipelining one may ask a server to
> HEAD
> (like stat) 10,000 files together, without having to wait for the
> responses.
> Over a high latency link (e.g. Australia -> USA), this might save perhaps
> an
> hour of waiting.
>

Under certain situations, 9p can do some forms of pipelining.  The tagged
requests don't have to be waited on in order, for the next outgoing request
to be sent, unless there's a dependency of one completing before the other,
or the evaluation of completion of a previous one on another.


>
> Such an asyncronous interface might be useful even when accessing local
> disks -
> if the filesystem receives 100 open/read/stat requests bundled together, it
> might optimise disk access to minimise seeking, as is commonly done for
> writes.
>
> By the way, I read the other day on this list that there is no need to
> improve
> cat(1).  Well for me, I still feel that the command `cat` without args
> should
> concatenate 0 files (producing no output), not copy stdin to stdout!
>

That's an interesting point of view.  I think the concept of "standard
input" is that if no input is given, it was going to be the fallback.  Same
goes for standard output.

With that said, I think cat is behaving just fine to take no arguments and
then default to the standard input and output :-).


>
>
> Sam
>
>
>

[-- Attachment #2: Type: text/html, Size: 2836 bytes --]

Re: [9fans] 9p vs http

[Venkatesh Srinivas]

> Under certain situations, 9p can do some forms of pipelining.  The tagged
> requests don't have to be waited on in order, for the next outgoing request
> to be sent, unless there's a dependency of one completing before the other,
> or the evaluation of completion of a previous one on another.

Only if the file & file server involved are decent.

-- vs

Re: [9fans] 9p vs http

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

On Mon, Nov 15, 2010 at 7:55 AM, Venkatesh Srinivas <me@acm.jhu.edu> wrote:

> > Under certain situations, 9p can do some forms of pipelining.  The tagged
> > requests don't have to be waited on in order, for the next outgoing
> request
> > to be sent, unless there's a dependency of one completing before the
> other,
> > or the evaluation of completion of a previous one on another.
>
> Only if the file & file server involved are decent.
>
> -- vs
>
> Yes.  I think that's the definition of "decent", but I'm not really up to
date on that enough to converse about decency...

[-- Attachment #2: Type: text/html, Size: 936 bytes --]

Re: [9fans] 9p vs http

[roger peppe]

2010/11/15 C H Forsyth <forsyth@vitanuova.com>:
>>% ls -lqd /n/dump/2006/0707/usr/rog
>>(000000000003d540 1122 80) d-rwxr-xr-x M 42850 rog rog 0 Jun  7  2005
>>/n/dump/2005/0707/usr/rog
>>% ls -lqd /n/dump/2006/0707/usr/rog
>>(000000000003d540 1157 80) d-rwxr-xr-x M 42850 rog rog 0 Jun 12  2006
>>/n/dump/2006/0707/usr/rog
>>they have the same qid but they're different directories
>>with different contents.
>
> the qid values are actually different

true, but qid.version doesn't help much.

Re: [9fans] 9p vs http

[C H Forsyth]

>% ls -lqd /n/dump/2006/0707/usr/rog
>(000000000003d540 1122 80) d-rwxr-xr-x M 42850 rog rog 0 Jun  7  2005
>/n/dump/2005/0707/usr/rog
>% ls -lqd /n/dump/2006/0707/usr/rog
>(000000000003d540 1157 80) d-rwxr-xr-x M 42850 rog rog 0 Jun 12  2006
>/n/dump/2006/0707/usr/rog
>they have the same qid but they're different directories
>with different contents.

the qid values are actually different

Re: [9fans] 9p vs http

[erik quanstrom]

> > the qid values are actually different
>
> true, but qid.version doesn't help much.

what!?  i'd hate to see a file server that didn't
much care which qid.version it had.  those directories
you listed are different.

- erik

Re: [9fans] 9p vs http

[roger peppe]

On 15 November 2010 16:48, erik quanstrom <quanstro@quanstro.net> wrote:
>> > the qid values are actually different
>>
>> true, but qid.version doesn't help much.
>
> what!?  i'd hate to see a file server that didn't
> much care which qid.version it had.  those directories
> you listed are different.

if you mount onto one, you'll see the mounted files
on the other.

gorka was talking about identifying files from their qid.
the version number doesn't help in identifying the file -
someone could have modified the file 35 times between
stats.

anyway, even if the version number is the same, the contents
may be different.

Re: [9fans] 9p vs http

[Russ Cox]

You're way off about the merits of pipelining.

As far as parallel requests are concerned,
the 9P protocol beats HTTP hands down
(as does basically any other request
response protocol), because it has explicit
unique IDs on the requests and responses.
That allows a server to respond to two
requests in an order different than their
arrival order.  If you send an expensive
HTTP request followed by a cheap one,
the server can't tell you the cheap answer
until it has told you the expensive answer.

The word "pipelining" means "we forgot to
put request and response IDs into the protocol
so we kludged it in by requiring responses to
happen in the same order as requests."
If a protocol says it has pipelining, that's
always a bad sign.

Russ

Re: [9fans] 9p vs http

[Dan Adkins]

That brings up a question of interest to me.  How do you effectively
read ahead with the 9p protocol?  Even if you issued many read
requests in parallel, the server is allowed to return less data than
was asked for.  You'll end up with holes in your buffer that require
at least another roundtrip to fill.

-Dan

Re: [9fans] 9p vs http

[erik quanstrom]

> if you mount onto one, you'll see the mounted files
> on the other.
>
> gorka was talking about identifying files from their qid.
> the version number doesn't help in identifying the file -
> someone could have modified the file 35 times between
> stats.

what definition of "file" are you using?  you've rejected the
definition that "file" is identified uniquely by path.  you've
also rejected the definition that file is identified uniquely
by qid.

> anyway, even if the version number is the same, the contents
> may be different.

i claim that a fs with this behavior would be broken.  intro(5)
seems to agree with this claim, unless i'm misreading.

- erik

Re: [9fans] 9p vs http

[roger peppe]

On 15 November 2010 19:29, erik quanstrom <quanstro@labs.coraid.com> wrote:
>> if you mount onto one, you'll see the mounted files
>> on the other.
>>
>> gorka was talking about identifying files from their qid.
>> the version number doesn't help in identifying the file -
>> someone could have modified the file 35 times between
>> stats.
>
> what definition of "file" are you using?  you've rejected the
> definition that "file" is identified uniquely by path.

i've only rejected that definition based on observation - intro(5)
does state that the qid path should be unique amongst all files
in the hierarchy, but that's a difficult ideal to live up to
when you're multiplexing filesystems.

>> anyway, even if the version number is the same, the contents
>> may be different.
>
> i claim that a fs with this behavior would be broken.  intro(5)
> seems to agree with this claim, unless i'm misreading.

you're right - fossil is broken in this respect, as is exportfs
{cd /mnt/term/dev; ls -lq | sort} for a quick demo.

the only correct way that i can see for a multiplexing fs to avoid this breakage
is the keep track of every qid that it sees, allocating a new qid of its
own for each one. no-one does this because it can use unbounded
memory. instead they map qids on open, because relatively
few files are opened.

that's why i was suggesting a "big enough" qid space as another
possibility. an alternative might be to remove qids from the
Dir structure entirely, returning them only on walk and open.

Re: [9fans] 9p vs http

[Yaroslav]

> That brings up a question of interest to me.  How do you effectively
> read ahead with the 9p protocol?  Even if you issued many read
> requests in parallel, the server is allowed to return less data than
> was asked for.  You'll end up with holes in your buffer that require
> at least another roundtrip to fill.

The option is to make servers obey R order for Ts with same tag - just
as Russ (right?) proposed elsewhere on the list.

Re: [9fans] 9p vs http

[erik quanstrom]

> The option is to make servers obey R order for Ts with same tag - just
> as Russ (right?) proposed elsewhere on the list.

no.  tags have no order and clients are specificly disallowed
from having multiple messages with the same tag outstanding.
again, see intro(5).

the option is to issue many Tread/Twrite messages concurrently,
say from the mount driver.  not all file servers will be able to deal with this,
so it'd need to be a mount flag.

- erik

Re: [9fans] 9p vs http

[erik quanstrom]

> > i claim that a fs with this behavior would be broken.  intro(5)
> > seems to agree with this claim, unless i'm misreading.
>
> you're right - fossil is broken in this respect, as is exportfs
> {cd /mnt/term/dev; ls -lq | sort} for a quick demo.

so what's fossil's excuse?

- erik

Re: [9fans] 9p vs http

[roger peppe]

On 16 November 2010 01:18, erik quanstrom <quanstro@quanstro.net> wrote:
>> > i claim that a fs with this behavior would be broken.  intro(5)
>> > seems to agree with this claim, unless i'm misreading.
>>
>> you're right - fossil is broken in this respect, as is exportfs
>> {cd /mnt/term/dev; ls -lq | sort} for a quick demo.
>
> so what's fossil's excuse?

i'd say it's a bug. fossil could easily reserve some number of bits
of the qid (say 20 bits) to make the files in the dump unique
while still allowing space for a sufficient number of live files.

that doesn't invalidate my original point though.

Re: [9fans] 9p vs http

[Russ Cox]

On Mon, Nov 15, 2010 at 2:00 PM, Dan Adkins <dadkins@gmail.com> wrote:
> That brings up a question of interest to me.  How do you effectively
> read ahead with the 9p protocol?  Even if you issued many read
> requests in parallel, the server is allowed to return less data than
> was asked for.  You'll end up with holes in your buffer that require
> at least another roundtrip to fill.

The traditional "store data" file servers that Unix users
would recognize tend to follow the Unix semantics that
a read will return as much data as it can.  If you're talking
to one of them you issue reads for 8k each or whatever
and then do the roundtrip if necessary, but it rarely is.

Russ

Re: [9fans] 9p vs http

[Charles Forsyth]

>i'd say it's a bug. fossil could easily reserve some number of bits
>of the qid (say 20 bits) to make the files in the dump unique
>while still allowing space for a sufficient number of live files.

that's possibly closest to the intent of the qid discussion in intro(5), although
it's not clear that considers the possibility of aliases within a hiearchy,
where it might make more sense to have the Qid.path associated with the file,
regardless of how you got to it. even so, for dump it's not been a big problem
in practice only because one usually binds from the dump, not into it.

i'm sure that somewhere it was suggested that high order bits of Qid.path
should be avoided by file servers to allow for their use to make qids unique
but i haven't been able to find that.

Re: [9fans] 9p vs http

[erik quanstrom]

> i'm sure that somewhere it was suggested that high order bits of Qid.path
> should be avoided by file servers to allow for their use to make qids unique
> but i haven't been able to find that.

unfortunately, there's just not enough bits to easily export
(an export)+.

i wonder if there's some way to expose the structure of the
export to the client so it will notice the devices are different.

- erik

Re: [9fans] 9p vs http

[Charles Forsyth]

>unfortunately, there's just not enough bits to easily export
>(an export)+.

i think that works: it checks for clashes.

Re: [9fans] 9p vs http

[roger peppe]

On 16 November 2010 16:32, Charles Forsyth <forsyth@terzarima.net> wrote:
>>unfortunately, there's just not enough bits to easily export
>>(an export)+.
>
> i think that works: it checks for clashes.

only when a file is actually walked to.

of course, that's fine in practise - the only thing
that actually cares about qids is mount.
but it doesn't fit the spec.