[9fans] Regarding 9p based "protocols" message framing

[9fans] Regarding 9p based "protocols" message framing

[Ciprian Dorin Craciun]

    Hello all!

    I'm reading through the nice `factotum` paper [1], and the section
describing the `rpc` file behaviour (section "2.6. Factotum
transactions") states:

~~~~
Programs authenticate with factotum by writing a request to the rpc
file and reading back the reply; this sequence is called an RPC
transaction. Requests and replies have the same format: a textual verb
possibly followed by arguments, which may be textual or binary. [...]
Once started, an RPC conversation usually consists of a sequence of
read and write transactions.
[...]
PS -> FS: start proto=apop role=server
FS -> PS: ok
~~~~

    Now my puzzle was related to what does actually "sequence of read
and write transactions" mean. To be more clearly:
    * the cited paragraph clearly involves two protocols: one is the
factotum RPC protocol which is layered on-top of the 9p protocol;
    * if we take only the factotum RPC protocol (in abstraction of 9p)
we **might** have the impression that each "request" and "response" is
sent as an individual message whose framing is that of an text line,
and 9p is used merely as a transport; (but this framing is not clearly
expressed, see below...);
    * thus if we take only this factotum RPC protocol into
consideration the implementor might have the impression that the
implementation should roughly be: keep receiving write requests until
we have an entire request (pseudo-C):
~~~~
handle_write (context, data) {
  append (context->buffer, data);
  if (request_ready (context) {
    handle_request (context);
  }
  return (succeed);
~~~~
    * but if we look at how it is implemented [2] it is actually: one
request and reply must map over exactly one 9p read or write
operation.
~~~~
handle_write (context, data) {
  if (parse_request (context, data))
    return (handle_request (context));
  else
    return (failed);
~~~~

    Why was I puzzled: because as a non Plan9 user / developer, I
usually think of the underlaying transport technology (be it sockets
or 9p) as a stream of bytes without explicit framing. Meanwhile I
think that the Plan9 usage of 9p is:
    * sometimes as a bidirectional message pipe (i.e. read == pull
message, write == push message, similar to what ZeroMQ library
offers);
    * sometimes as an stateless RPC (like HTTP): open file, write
request data in maybe multiple write operations, read reply data in
maybe multiple read operations; (like the `ctl` file, as I guess `cat
>ctl` makes one write per read line);
    * sometimes as an un-seekable stream of data (like the `log` file);

    Now I see here advantages and disadvantages... Advantages:
    * the developer doesn't need to bother anymore with framing -- it
is done directly by the FS; (and gosh is this a relief, as every other
protocol invented its own framing: length followed by data, line feed
terminated, empty line terminated, random token ended, etc.)
    * concurrent outstanding requests -- just call write in parallel;
(I've read the protocol and the specification doesn't say its
forbidden, but I've read something a few months ago on this mailing
list which suggested it should not be done???)

    Disadvantages:
    * the tools must be aware of this mapping between 1 request == 1
write operation; thus if for example my request would be composed of
multiple lines, and I use `cat >./some-file` as an "interface"; then
`cat` would read one line and immediately send it as an individual
write operation which of course is not respecting the "protocol" built
ontop of 9p;


    Thus I would like to hear the position of the people on this
mailing list about:
    * the explicitly mapping of one request to one 9p operation;
    * the "recommended" usage "patterns" of 9p when building "control"
protocols ontop;

    Thanks,
    Ciprian.


    [1] Security in Plan 9 -- http://plan9.bell-labs.com/sys/doc/auth.html
    [2] http://plan9.bell-labs.com/sources/plan9/sys/src/cmd/auth/factotum/rpc.c

Re: [9fans] Regarding 9p based "protocols" message framing

[Richard Miller]

>     * the tools must be aware of this mapping between 1 request == 1
> write operation; thus if for example my request would be composed of
> multiple lines, and I use `cat >./some-file` as an "interface"; then
> `cat` would read one line and immediately send it as an individual
> write operation which of course is not respecting the "protocol" built
> ontop of 9p;

In general, control requests are not multiple lines (I presume by "line"
you mean a string of characters delimited by '\n').  As you say, each
request must be presented in a single 9p message, and this fits well
with the guarantee that 9p read/write messages are atomic [provided
tbe length is <= iounit].  The first example in 2.6 of /sys/doc/auth is
misleading -- if you try to do literally what it says:

	% cd /mnt/factotum
	% cat >ctl
	key dom=bell-labs.com proto=p9sk1 user=gre
		!password='don''t tell'
	key proto=apop server=x.y.com user=gre
		!password='bite me'
	^D

you'll find you get a write error after the first line; each 'key'
command actually has to be sent as a single write.  You can't even
get it to accept multiple lines in one write:

	term% echo 'key dom=bell-labs.com proto=p9sk1 user=gre
			!password=bite-me' >/mnt/factotum/ctl
	echo: write error: multiline write not allowed

In my experience, sending control requests as single atomic writes
is a natural way to work.  With failed requests communicated as
write errors, this makes it simple to match the error status with
the request.

Multiline responses are somewhat more common, and there is some
inconsistency in whether responses also need to be read as a single
atomic operation.  This can make it tricky, for example, to use the
read(1) command -- which reads one char at a time -- in a script
dealing with a device control interface.

For example, this loop works:
	</dev/irqalloc while (x=`{read}) echo $x
but this doesn't:
	</dev/ioalloc while (x=`{read}) echo $x

Re: [9fans] Regarding 9p based "protocols" message framing

[Russ Cox]

On Fri, Mar 16, 2012 at 7:40 AM, Ciprian Dorin Craciun
<ciprian.craciun@gmail.com> wrote:
>  Why was I puzzled: because as a non Plan9 user / developer, I
> usually think of the underlaying transport technology (be it sockets
> or 9p) as a stream of bytes without explicit framing.

Even outside Plan 9, this is not always the case.
UDP, IP, ethernet, Unix datagram sockets, email, HTTP.
They're all message-based protocols.

Russ

Re: [9fans] Regarding 9p based "protocols" message framing

[Ciprian Dorin Craciun]

On Fri, Mar 16, 2012 at 19:36, Russ Cox <rsc@swtch.com> wrote:
> On Fri, Mar 16, 2012 at 7:40 AM, Ciprian Dorin Craciun
> <ciprian.craciun@gmail.com> wrote:
>>  Why was I puzzled: because as a non Plan9 user / developer, I
>> usually think of the underlaying transport technology (be it sockets
>> or 9p) as a stream of bytes without explicit framing.
>
> Even outside Plan 9, this is not always the case.
> UDP, IP, ethernet, Unix datagram sockets, email, HTTP.
> They're all message-based protocols.
>
> Russ


    I am aware that there are other types of transports with other
semantics. Nevertheless I guess that at least 90% of the existing
protocols are built directly on top of stream based transports than
datagram ones. (I'm not speaking of ubiquity / underlaying technology,
or else we'd conclude that we use Ethernet in 99.999% of the cases.)
And what is even more funny, is that I guess a good majority of the
protocols are in the end message-based (as you've mentioned HTTP,
SMTP, etc.) although they are built ontop of a stream based protocol.

    Furthermore 9p exposes a semantic more similar with TCP than with
UDP, and only when you take into account the read / write atomicity
you see it as a datagram protocol.

    My concert was that in general a programmer used to implement
things ontop of stream based protocols was "thought" to expect that
one read on the client is not in general one read on the server due to
fragmentation, etc., and that he must frame his messages accordingly.

    Thus my question was if when working with 9p it is a good choice
to do explicit framing (as in the case building ontop of TCP) or to
take for granted the read / write atomicity.

    Ciprian.

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

On Fri Mar 16 13:37:36 EDT 2012, rsc@swtch.com wrote:
> On Fri, Mar 16, 2012 at 7:40 AM, Ciprian Dorin Craciun
> <ciprian.craciun@gmail.com> wrote:
> >  Why was I puzzled: because as a non Plan9 user / developer, I
> > usually think of the underlaying transport technology (be it sockets
> > or 9p) as a stream of bytes without explicit framing.
> 
> Even outside Plan 9, this is not always the case.
> UDP, IP, ethernet, Unix datagram sockets, email, HTTP.
> They're all message-based protocols.

of course inside plan 9, pipes are repect write boundaries, too.

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[Yaroslav]

>  Why was I puzzled: because as a non Plan9 user / developer, I
> usually think of the underlaying transport technology (be it sockets
> or 9p) as a stream of bytes without explicit framing.

As I understand, 9P itself is designed to operate on top of a
message-oriented transport; however, it has everything required to run
over a stream, esp. message length at beginning of every message.
Framing is done by the library: the read9pmsg routine performs as many
reads as necessary to return a complete 9P message to the caller.

Re: [9fans] Regarding 9p based "protocols" message framing

[Dan Cross]

On Tue, Mar 20, 2012 at 7:42 AM, Yaroslav <yarikos@gmail.com> wrote:
>>  Why was I puzzled: because as a non Plan9 user / developer, I
>> usually think of the underlaying transport technology (be it sockets
>> or 9p) as a stream of bytes without explicit framing.
>
> As I understand, 9P itself is designed to operate on top of a
> message-oriented transport; however, it has everything required to run
> over a stream, esp. message length at beginning of every message.
> Framing is done by the library: the read9pmsg routine performs as many
> reads as necessary to return a complete 9P message to the caller.

Perhaps initially: over an IP network, 9P used to run over IL.  With
9P2000, IL was deprecated and 9P was most typically run over TCP.
There was a very old message to 9fans (like, early 90's kind of old)
that implied that IL was much more efficient than TCP on the wire, but
it probably doesn't matter now.  9P itself is not a stream-oriented
protocol, nor is it what one would generally call, 'transport
technology.'

        - Dan C.

Re: [9fans] Regarding 9p based "protocols" message framing

[Ciprian Dorin Craciun]

On Tue, Mar 20, 2012 at 14:32, Dan Cross <crossd@gmail.com> wrote:
> 9P itself is not a stream-oriented
> protocol, nor is it what one would generally call, 'transport
> technology.'


    I would beg to differ on this subject... Because a lot of tools in
the Plan9 environment expose their facilities as 9p file systems, but
expose other semantics than that of "generic" files -- i.e. a
contiguous stream of bytes from start to EOF -- like for example RPC
semantic in case of factotum; thus I would say that 9p is used as a
"session" layer in the OSI terminology. (But as in TCP/IP stack we
don't have other layers between "transport" and "application" I would
call it a "transport" layer in such a context.)

Re: [9fans] Regarding 9p based "protocols" message framing

[Dan Cross]

On Tue, Mar 20, 2012 at 4:30 PM, Ciprian Dorin Craciun
<ciprian.craciun@gmail.com> wrote:
> On Tue, Mar 20, 2012 at 14:32, Dan Cross <crossd@gmail.com> wrote:
>> 9P itself is not a stream-oriented
>> protocol, nor is it what one would generally call, 'transport
>> technology.'
>
>    I would beg to differ on this subject... Because a lot of tools in
> the Plan9 environment expose their facilities as 9p file systems, but
> expose other semantics than that of "generic" files -- i.e. a
> contiguous stream of bytes from start to EOF -- like for example RPC
> semantic in case of factotum; thus I would say that 9p is used as a
> "session" layer in the OSI terminology. (But as in TCP/IP stack we
> don't have other layers between "transport" and "application" I would
> call it a "transport" layer in such a context.)

That's one way of looking at it.  However, the "file as a stream of
bytes" abstraction is mapped onto 9P at a higher layer; 9P itself is
really about discrete messages.  The canonical "transport" layer in
TCP/IP is TCP.

But we're arguing semantics at that point; regardless, I think you'd
find you hold something of a minority view.

        - Dan C.

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

> >    I would beg to differ on this subject... Because a lot of tools in
> > the Plan9 environment expose their facilities as 9p file systems, but
> > expose other semantics than that of "generic" files -- i.e. a
> > contiguous stream of bytes from start to EOF -- like for example RPC
> > semantic in case of factotum; thus I would say that 9p is used as a
> > "session" layer in the OSI terminology. (But as in TCP/IP stack we
> > don't have other layers between "transport" and "application" I would
> > call it a "transport" layer in such a context.)
> 
> That's one way of looking at it.  However, the "file as a stream of
> bytes" abstraction is mapped onto 9P at a higher layer; 9P itself is
> really about discrete messages.  The canonical "transport" layer in
> TCP/IP is TCP.

i think you can put a finer point on this.  section 5 of the manual
defines 9p by defining its messages, and the relationship between them.
what it doesn't define is semantics.

what does it mean to "write n bytes at offset o".  well, nothing in particular,
other than you send the appropriate Twrite and get the matching Rwrite back.
it's perfectly valid for the file server to ignore the write (/dev/null) or to
halt and catch fire.  file servers as a whole are unfortunately rather more
ordinary.

i think it's important to note (see /sys/doc/net.ps) that this would all
be utter chaos if it weren't for conventions.  plan 9 has strong conventions,
but there aren't any cops.

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

> Perhaps initially: over an IP network, 9P used to run over IL.

still does, including on the system i'm sending this from.

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[Yaroslav]

>> Perhaps initially: over an IP network, 9P used to run over IL.
>
> still does, including on the system i'm sending this from.
>

What advantages does it have over TCP? Does it worths the effort of
tailoring it back in, esp. for a fossil/venti site?

--
- Yaroslav

Re: [9fans] Regarding 9p based "protocols" message framing

[Anthony Sorace]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]

> What advantages does it have over TCP?

Significantly simpler implementation.
Lower overhead, giving better speed over local links.

Also, although not an advantage of IL per se, there are a few
advantages to Plan 9 having another protocol in the stack,
especially one which can run 9p. It helps guard against any
accidental network-dependent creep.

> Does it worths the effort of tailoring it back in,

It's not hard to get back in. Erik's 9atom has it. Whether it's
worth it or not is hard to say. I like having it, but NAT boxes
(and occasionally non-NAT routers, although that's more
rare) can cause problems. It is slower over common wide-
area links (lacking TCP's congestion control).

> esp. for a fossil/venti site?

It's mandatory if you want to use kenfs; it's entirely optional
in a fossil/venti (or cwfs or kfs or whatever) environment. It's
sort of depends what you want from the system.

Personally, I'd suggest doing it if you have multiple systems
on the same local net for the pedagogical benefits alone.

Anthony


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 210 bytes --]

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

> > What advantages does it have over TCP?
>
> Significantly simpler implementation.
> Lower overhead, giving better speed over local links.

it's also message based.  so things like rx depend on a protocol
like il that preserves 0-byte writes.

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

On Wed Mar 21 16:11:41 EDT 2012, yarikos@gmail.com wrote:
> >> Perhaps initially: over an IP network, 9P used to run over IL.
> >
> > still does, including on the system i'm sending this from.
> >
>
> What advantages does it have over TCP? Does it worths the effort of
> tailoring it back in, esp. for a fossil/venti site?

it takes maybe 10 minutes to put it back in.  9atom has il still,
since i still use it.  i never took it out.

i see why one would want to eliminate il and go all tcp, but
il is simplier and has no problems that bother me operationally.
also there are some fine points of the argument where i have
a different opinion than some.  of course i may also be motivated
by not wanting to put tcp in the file server.  ymmv.

(in fact, i'd go further than il.  why do we need the ip in il?)

tcp otoh has a handful of bugs that bite me now and again.  the
biggest current one is that there appears to be a bug in the
implementation that triggers a condition that looks as if nagle
is disabled for some connections and early bytes in the window
are missing. (infinite resequence queue, full of tinygrams.)
this crashes boxes when the run out of kernel memory.

i've worked around this by limiting the resequence queue to
a reasonable (scaled with the window) number of packets as
well as bytes.

clearly this is just a hack.  but until the trigger for this condition
is understood, it'll have to do.

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[Anthony Sorace]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

On Mar 21, 2012, at 16:27 , erik quanstrom wrote:

> (in fact, i'd go further than il.  why do we need the ip in il?)

I'd love to have an IP-free, 9p-capable network in the mix. Plan 9
had this, way back when, with "nonet" (?), but that didn't survive
the removal of the streams system in the 3rd edition (I know you
know this already,but I'm not sure how widely it's known).

Anthony


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 210 bytes --]

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

> I'd love to have an IP-free, 9p-capable network in the mix. Plan 9
> had this, way back when, with "nonet" (?), but that didn't survive
> the removal of the streams system in the 3rd edition (I know you
> know this already,but I'm not sure how widely it's known).

i'm probablly missing something, but i don't see why your lan couldn't
be ip-free.  you may have to have a sacrifical super-nat machine that
transplants, say, ssh/nonet onto ssh/ip.  the more plan 9-y way would
be to import the gateway.  i'm not sure which would be more elegant.

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[cinap_lenrek]

importing /net is possible.

just recently replaced openwrt linux nat router with a plan9 machine
importing its /net.alt to avoid problems with random tcp connection
drops.

wrote a simple socks server so windows can use it as a gateway too.
even use bittorrent thru that thing with udp relay :)

works nicely overall. some programs have problems with /net.alt tho.
mainly ape programs wich require a "bind /net.alt /net". wrote hg
extension that hooks mercurials http "object obfuscated interface"
and uses plan9 webfs (/mnt/web) instead, also makes hg use factotum
indirectly thru webfs. only disadvantage is that one doesnt get
proper errstr thru all that python/ape wrappery :(

--
cinap

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

> just recently replaced openwrt linux nat router with a plan9 machine
> importing its /net.alt to avoid problems with random tcp connection
> drops.

where's it importing /net.alt from?

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[cinap_lenrek]

gateway machine has two ethernet cards. /net.alt is the external
internet facing side where i get ip over dhcp from cable modem.
the /net is the internal local network.

running "import gateway /net.alt" on random plan9 machine to import
the internet.

--
cinap

Re: [9fans] Regarding 9p based "protocols" message framing

[cinap_lenrek]

http://swtch.com/cgi-bin/plan9history.cgi?f=1990/1210/port/devnonet.c;v=diff;e=1

(from http://softwaremagpie.blogspot.de/2008/04/fistful-of-protocols.html)

--
cinap

Re: [9fans] Regarding 9p based "protocols" message framing

[erik quanstrom]

thanks, that's great.

the retransmission interval is probally off by 3/4
orders of magnitude for today's local networks.
50-500µs would be a decent static retransmit interval.

(goes to show that timing sleeps in ms anachronistic.)

- erik

Re: [9fans] Regarding 9p based "protocols" message framing

[David du Colombier]

> it takes maybe 10 minutes to put it back in.

10 seconds ☺

term% cd /
term% 9fs sources
term% ape/patch -p1 < /n/sources/contrib/djc/il/il-plan9.diff
term% ape/patch -p1 < /n/sources/contrib/djc/il/il-netlog.diff
term% ape/patch -p1 < /n/sources/contrib/djc/il/il-conf.diff

-- 
David du Colombier