[9fans] some Plan9 related ideas

[9fans] some Plan9 related ideas

[Bhanu Nagendra Pisupati]

Hello,
I am a PhD student at Indiana University working on the application of
some Plan 9 related to the embedded domain. Specifically I am looking at
embedded debugging, configuration and device management using the
distributed virtual filesystem model. I have looked at a few
different types of embedded systems including multi board systems
containing SoC's (system on chips) and sensor networks. I have come up
with a few different ideas (listed below) to address as part of my thesis
work regarding which I hope to get feedback from the wider Plan 9
community. Any thoughts and opinions are greatly appreciated.
Thanks in advance,
-Bhanu

==========================================================================

* Downloadable namespaces
The namespace supported by a filesystem can be modified (possibly using a
configuration file provided by the filesystem) to correspond to changes in
system that the filesystem in encapsulating. For instance, if the
filesystem was to abstract a set of sensor networks, then at start up
time the user would describe the layout of the network to the
filesystem possibly by writing it as an XML description to the
configuration file and
modify(initialize) the namespace. If the layout of the
network changes in due course, the namespace could be similarly modified
by using the configuration file to reflect the change.

The basic idea is to be able to describe and modify the layout of the
namespace of a filesystem namespace dynamically.

* 'Tailcall optimizations' for filesystems with other mounted filesystems

Consider a filesystem with the layout shown below:

            FS1
          /  |  \
         f1  f2  FS2
                 / \
                f3  f4

 Filesystem FS1 contains 2 files (f1 & f2) locally apart from
having a mounted filesystem FS2. To a client that mounts FS1, the fact
that it has a mounted filesystem (FS2) is transparent. When a client tries
to access file in the mounted filesystem (f3 in FS2) then, FS1 passes the
request on to FS2 which processes
the request hands the result back to FS1 which then returns the result to
the client. However this operation could be made more efficient if FS2
could be made to return the result directly to the client rather than
sending it to an upstream filesystem. This is analogous to the tail call
optimization in compilers where a function call made in the tail position
of a subroutine returns directly to the original caller rather than
returning to the subroutine and then having it return to the caller.
The situation obviously get progressively worse as the number of mounted
filesystems in a chain get longer.

This idea is based on the VMTP protocol.

* Macro messages
Lightweight clients (such as microcontrollers) that communicate with a
fileserver using 9P protocol over flaky radio connections would benefit
from being able to compose several messages (eg: OPEN+READ+CLUNK)
together a a single macro packet. This because being able to send one
larger packet takes much lesser power than taking multiple smaller
packets. Also when multiple devices send data over radio, getting
access to a free time slot to communicate is hard. So it makes sense to
limit the number of occasions when messages have to be sent. Also if
in most cases, the number of operations performed during the time a file
is open are small, it limits the number of open files and corresponding
the state that needs to be stored for fids.

* Stateless variants of 9P
This is more hypothetical, but the basic idea is to design and use a
variant of 9P which is stateless (or uses little state) and hence
better suited for use on devices with little RAM

==========================================================================

Re: [9fans] some Plan9 related ideas

[Sape Mullender]

> I am a PhD student at Indiana University working on the application of
> some Plan 9 related to the embedded domain. Specifically I am looking at
> embedded debugging, configuration and device management using the
> distributed virtual filesystem model. I have looked at a few
> different types of embedded systems including multi board systems
> containing SoC's (system on chips) and sensor networks. I have come up
> with a few different ideas (listed below) to address as part of my thesis
> work regarding which I hope to get feedback from the wider Plan 9
> community. Any thoughts and opinions are greatly appreciated.

That's great.  At the labs we're also using Plan 9 as an embedded operating
system and we're demonstrating to the old-fashioned  folks who believe
in dedicated embedded systems from manufacturares that shall remain nameless
that we can get things going reliably in a fraction of the time.

> * Downloadable namespaces
> The namespace supported by a filesystem can be modified (possibly using a
> configuration file provided by the filesystem) to correspond to changes in
> system that the filesystem in encapsulating.

Right now, Plan 9 interprets the file /lib/namespace when it starts up
(and when it starts up applications that respond to external requests).
That describes a tailored — but not dynamic — name space already.

> For instance, if the
> filesystem was to abstract a set of sensor networks, then at start up
> time the user would describe the layout of the network to the
> filesystem possibly by writing it as an XML description to the
> configuration file and
> modify(initialize) the namespace. If the layout of the
> network changes in due course, the namespace could be similarly modified
> by using the configuration file to reflect the change.

XML?  You must like overkill.  First question is, does each node have it's
own local file system or does it get its files from a file server?  If the latter,
I can imagine building a file system that serves different nodes with different,
tailored and dynamic, content.  If the former, I can imagine a distributed
implementation of such a service.  If your name space is highly dynamic,
mounting and unmounting may not be the most efficient or flexible path.
A file server that adapts the files it serves to the sensor node's needs may
be a more powerful way, as well as one that is easier to realize.

	Sape
	

Re: [9fans] some Plan9 related ideas

[Francisco Ballesteros]

> > * Downloadable namespaces
> > The namespace supported by a filesystem can be modified (possibly using a
> > configuration file provided by the filesystem) to correspond to changes in
> > system that the filesystem in encapsulating.
> 
> Right now, Plan 9 interprets the file /lib/namespace when it starts up
> (and when it starts up applications that respond to external requests).
> That describes a tailored — but not dynamic — name space already.

Plan B changes this to provide dynamic name spaces, where you request
resources but do not directly mount them. See at lsub.org info about Plan B.
In a few days, I'll be implementing this service as a user level
process in Plan 9
(following  a suggestion from rsc). In fact, this is part of our effort
 to integrate our system services back into Plan 9.

For example, we are using RFID and X10 servers (which are small enough that
you might embed them). Our client machines run:
     mount -U  /devs/x10!L136 /n/x10
to request mounting a *U*nion of file trees at /n/x10 for any server
discovered, e.g.,  with name /devs/x10 and location "136"

Is this what you wanted? If so, our modified Plan 9 (i.e. Plan B)
already provides
that.

As Sape said, XML  would be an overkill (though it might be easier
to publish ;-)) ).  We use the "/devs/x10!L136!Unemo..." strings to
specify things.

Re: [9fans] some Plan9 related ideas

[Francisco Ballesteros]

Forgot to say, your "layout", we build by mounting whatever we want.
We can mount all the devices at a single directory, or group them by
location:

for (l in $locations)
    mount -U /devs/x10!L$l /n/x10$l

You can always automate this, thus there is no need for force or write
a particular spec.

[9fans] Re: some Plan9 related ideas

[Dave Eckhardt]

> 'Tailcall optimizations' for filesystems with other mounted
> filesystems

In Plan 9 all mounts are done at the client side, so this
wouldn't be an optimization--it's the only case.

> * Macro messages
> Lightweight clients (such as microcontrollers) that communicate
> with a fileserver using 9P protocol over flaky radio connections
> would benefit from being able to compose several messages (eg:
> OPEN+READ+CLUNK) together a a single macro packet.

9P runs over TCP, so I don't think there's a packet-boundary
problem here.  Getting a client to send the three messages
at once would seem to be the problem, since at present the
open() system call won't complete until the 9P message gets
its reply... there isn't an open()+read()+close() system
call.  And you'd probably need to reinvent or clone existing
work on batch RPC's to do things like fill the result of the
OPEN into the subsequent READ request.

> Also if in most cases, the number of operations performed
> during the time a file is open are small, it limits the number
> of open files and corresponding the state that needs to be
> stored for fids.

Collecting or finding somebody else's data on that "if" might
be a good first step.

Dave Eckhardt

Re: [9fans] Re: some Plan9 related ideas

[Francisco Ballesteros]

We added two (library) calls readf and writef that perform file I/O
besides resolving the name. We found ourselves calling them a lot, because
in many cases it's very convenient. They would be
an opportunity to "batch" walk/open/read(s)/clunk, which happen
a lot. However, this would require changing the kernel (even more than we
did for Plan B).

I think the man page is at http://planb.lsub.org/magic/man2html/2/readf

We would be very interested on implementing this change for Plan 9
because we found
that the main problem we have in Plan B
is  the latency accessing the various file servers,
because of the addition of the individual RPC  turn around times.

Any thought on this? 

On 8/30/05, Dave Eckhardt <davide+p9@cs.cmu.edu> wrote:
> > 'Tailcall optimizations' for filesystems with other mounted
> > filesystems
> 
> In Plan 9 all mounts are done at the client side, so this
> wouldn't be an optimization--it's the only case.
> 
> > * Macro messages
> > Lightweight clients (such as microcontrollers) that communicate
> > with a fileserver using 9P protocol over flaky radio connections
> > would benefit from being able to compose several messages (eg:
> > OPEN+READ+CLUNK) together a a single macro packet.
> 
> 9P runs over TCP, so I don't think there's a packet-boundary
> problem here.  Getting a client to send the three messages
> at once would seem to be the problem, since at present the
> open() system call won't complete until the 9P message gets
> its reply... there isn't an open()+read()+close() system
> call.  And you'd probably need to reinvent or clone existing
> work on batch RPC's to do things like fill the result of the
> OPEN into the subsequent READ request.
> 
> > Also if in most cases, the number of operations performed
> > during the time a file is open are small, it limits the number
> > of open files and corresponding the state that needs to be
> > stored for fids.
> 
> Collecting or finding somebody else's data on that "if" might
> be a good first step.
> 
> Dave Eckhardt
>

Re: [9fans] Re: some Plan9 related ideas

[Russ Cox]

I don't know whether the change is worth doing, but
here is a simple way to do it.  Define that a client may
send more than one message with the same tag, and
in that case servers must process those messages
sequentially.  This is not very hard to implement on the
server side, and the single-threaded servers needn't
change at all.

Now to implement the so-called batch RPC, you just
send three messages in a row with the same tag:

  tag Topen fid name mode
  tag Twrite fid offset data
  tag Tclunk fid

and then wait for three responses to come back.
Since the client has complete control over the choice of
tags and fids, there is no information in the R messages
needed to generate the T messages.  The various results
are completely distinguishable: on success you get
back Ropen, Rwrite, Rclunk,  If the Topen fails, then you'll
get back Rerror, Rerror (unknown fid), Rclunk.  If the Twrite
fails you'll get Ropen, Rerror, Rclunk.

I have no idea whether this is worth doing.  My gut reaction
is no, but maybe someone will prove me wrong.  My point
is only that the protocol need hardly change.

Russ

[9fans] tcs bug

[arisawa]

Hello,

tcs both for plan 9 and for unix has a bug in reading utf text.
that comes from:
utf_in(int fd, long *notused, struct convert *out){
     char buf[N];
     ...
     while((n = read(fd, buf+tot, N-tot)) >= 0){
         ...
}

in utf.c

N is assigned to be 10000 in hdr.h

if you set N to 10, you will find the problem more clearly:
tcs cannot handle correctly utf character boundary.

for example, assume a.txt have the content:
aaaaaaaこの

term% xd -c a.txt
0000000   a  a  a  a  a  a  a e3 81 93 e3 81 ae \n
000000e

tcs can handle this text because N=10 is just uft boundary
but tcs fails if 'a' are 6 or 8 ...

tcs is very important for me.
Who maintains tcs ?
I might help debugging.

Kenji Arisawa

Re: [9fans] tcs bug

[Rob Pike]

ah yes, the dreaded partial rune problem. lots of programs
must cope with this issue.

-rob

On 8/31/05, arisawa@ar.aichi-u.ac.jp <arisawa@ar.aichi-u.ac.jp> wrote:
> Hello,
> 
> tcs both for plan 9 and for unix has a bug in reading utf text.
> that comes from:
> utf_in(int fd, long *notused, struct convert *out){
>      char buf[N];
>      ...
>      while((n = read(fd, buf+tot, N-tot)) >= 0){
>          ...
> }
> 
> in utf.c
> 
> N is assigned to be 10000 in hdr.h
> 
> if you set N to 10, you will find the problem more clearly:
> tcs cannot handle correctly utf character boundary.
> 
> for example, assume a.txt have the content:
> aaaaaaaこの
> 
> term% xd -c a.txt
> 0000000   a  a  a  a  a  a  a e3 81 93 e3 81 ae \n
> 000000e
> 
> tcs can handle this text because N=10 is just uft boundary
> but tcs fails if 'a' are 6 or 8 ...
> 
> tcs is very important for me.
> Who maintains tcs ?
> I might help debugging.
> 
> Kenji Arisawa
> 
>

Re: [9fans] Re: some Plan9 related ideas

[Uriel]

I was reading old archives, and I'm probably a bit dense; but what is
the reason to use the same tag for the three messages?

Wouldn't that be able to break a server that expected tags not to be
reused until the corresponding Rmessage had been sent? Or I'm probably
misunderstanding how the tag is used...

And what are the kernel changes nemo mentioned?

I have to add that this is a big deal for connections with latencies 
of >100ms. 

I was also thinking of some library that would make it easier to add
fcp(1)-like functionality to other apps, but that is not very general.

uriel

On Tue, Aug 30, 2005 at 01:46:40PM -0400, Russ Cox wrote:
> I don't know whether the change is worth doing, but
> here is a simple way to do it.  Define that a client may
> send more than one message with the same tag, and
> in that case servers must process those messages
> sequentially.  This is not very hard to implement on the
> server side, and the single-threaded servers needn't
> change at all.
> 
> Now to implement the so-called batch RPC, you just
> send three messages in a row with the same tag:
> 
>   tag Topen fid name mode
>   tag Twrite fid offset data
>   tag Tclunk fid
> 
> and then wait for three responses to come back.
> Since the client has complete control over the choice of
> tags and fids, there is no information in the R messages
> needed to generate the T messages.  The various results
> are completely distinguishable: on success you get
> back Ropen, Rwrite, Rclunk,  If the Topen fails, then you'll
> get back Rerror, Rerror (unknown fid), Rclunk.  If the Twrite
> fails you'll get Ropen, Rerror, Rclunk.
> 
> I have no idea whether this is worth doing.  My gut reaction
> is no, but maybe someone will prove me wrong.  My point
> is only that the protocol need hardly change.
> 
> Russ

Re: [9fans] Re: some Plan9 related ideas

[Russ Cox]

> I was reading old archives, and I'm probably a bit dense; but what is
> the reason to use the same tag for the three messages?

The reason is you don't have to wait for the response to the first
before sending the second and third, avoiding two round trip times.

> Wouldn't that be able to break a server that expected tags not to be
> reused until the corresponding Rmessage had been sent?

Yes, but I did say I was redefining the protocol.  And single-threaded
servers (the majority of our servers by code volume) don't care.

    Define that a client may send more than one message
    with the same tag, and in that case servers must process
    those messages sequentially.  This is not very hard to
    implement on the server side, and the single-threaded
    servers needn't change at all.

Russ

Re: [9fans] Re: some Plan9 related ideas

[Uriel]

On Mon, Oct 17, 2005 at 07:23:39AM -0400, Russ Cox wrote:
> > I was reading old archives, and I'm probably a bit dense; but what is
> > the reason to use the same tag for the three messages?
> 
> The reason is you don't have to wait for the response to the first
> before sending the second and third, avoiding two round trip times.
Yes, but what I didn't understand is why you needed to use the same tag,
I thought you could do this without chaning the protocol.

After some discussion in #plan9 we guessed that the reason is threaded
servers...

Could you explain with more detail how it would work from the (threaded)
server POV? I was thinking that the server could use the fid to avoid
threads stepping into each other, and still avoid having to change the
protocol at all...

And I'm still curious what kernel changes nemo was talking about.

Sorry for being dense

uriel

Re: [9fans] Re: some Plan9 related ideas

[Russ Cox]

> > > I was reading old archives, and I'm probably a bit dense; but what is
> > > the reason to use the same tag for the three messages?

> > The reason is you don't have to wait for the response to the first
> > before sending the second and third, avoiding two round trip times.

> Yes, but what I didn't understand is why you needed to use the same tag,
> I thought you could do this without chaning the protocol.

I redefined use of the same tag to mean "you have to finish the
previous message with this tag before processing this message",
so that if you send a Topen followed by a Tread and the open
blocks for whatever reason (disk i/o, say), the remote server
doesn't try to run the Tread and send back a "fid not in use"
error or some such.

> Could you explain with more detail how it would work from the (threaded)
> server POV? I was thinking that the server could use the fid to avoid
> threads stepping into each other, and still avoid having to change the
> protocol at all...

The threaded server would just have a list of requests associated with
each tag instead of a single request.  When it finishes one it can move
on to the next.

Under the current protocol you are not allowed to send a Tread request
using a fid that the server has not acknowleged via Rattach or Ropen.
So your approach still requires redefining the protocol.  Also I might have
multiple I/Os going on and not care what order they get handled.
Synchronization based on the fid changes current situations.  Basing it
on the tag uses what were previously illegal situations.

> And I'm still curious what kernel changes nemo was talking about.

Read his post where he talks about mount -U.

If you mean readf and writef, those weren't kernel changes.
They were the obvious library wrappers.

Russ

Re: [9fans] Re: some Plan9 related ideas

[Uriel]

On Mon, Oct 17, 2005 at 09:03:22AM -0400, Russ Cox wrote:
> I redefined use of the same tag to mean "you have to finish the
> previous message with this tag before processing this message",
> so that if you send a Topen followed by a Tread and the open
> blocks for whatever reason (disk i/o, say), the remote server
> doesn't try to run the Tread and send back a "fid not in use"
> error or some such.
Yes, that would be a problem with threaded servers, as you indicated.
 
> > Could you explain with more detail how it would work from the (threaded)
> > server POV? I was thinking that the server could use the fid to avoid
> > threads stepping into each other, and still avoid having to change the
> > protocol at all...
> 
> The threaded server would just have a list of requests associated with
> each tag instead of a single request.  When it finishes one it can move
> on to the next.
> 
> Under the current protocol you are not allowed to send a Tread request
> using a fid that the server has not acknowleged via Rattach or Ropen.
Ah! I was not aware of this restriction.

> So your approach still requires redefining the protocol.  Also I might have
> multiple I/Os going on and not care what order they get handled.
> Synchronization based on the fid changes current situations.  Basing it
> on the tag uses what were previously illegal situations.
Good point. I understand better now the reasons for your approach.
 
> > And I'm still curious what kernel changes nemo was talking about.
> 
> Read his post where he talks about mount -U.
> 
> If you mean readf and writef, those weren't kernel changes.
> They were the obvious library wrappers.
Sorry, I was not clear enough, I was referring to this:

"We added two (library) calls readf and writef that perform file I/O
besides resolving the name. We found ourselves calling them a lot, because
in many cases it's very convenient. They would be
an opportunity to "batch" walk/open/read(s)/clunk, which happen
a lot. However, this would require changing the kernel (even more than we
did for Plan B)."

(BTW, I get the feeling that readf/writef might be convenient, but you could
easily end up sending superfluous walks/opens/clunks) 

And I'm not convinced by Plan B's style of 'one value per file' for (almost?)
everything, maybe it smells to me too much of linux's sysfs :)

(Nothing wrong with 'one value per file' where it makes sense, I just don't
think it works well as a general rule)

uriel

Re: [9fans] Re: some Plan9 related ideas

[Russ Cox]

> "We added two (library) calls readf and writef that perform file I/O
> besides resolving the name. We found ourselves calling them a lot, because
> in many cases it's very convenient. They would be
> an opportunity to "batch" walk/open/read(s)/clunk, which happen
> a lot. However, this would require changing the kernel (even more than we
> did for Plan B)."

Well once you've changed the protocol to accomodate some
form of batching, changes are good you'll have to change
the kernel to use it.

Russ