[9fans] 9P vs. FUSE

[9fans] 9P vs. FUSE

[Enrico Weigelt]

Hi folks,


did anyone do any comparisons on 9P vs. FUSE for (mostly-)local uses ? 

There're a lot of interesting FUSE-based filesystems out there,
and programming w/ FUSE seems to be quite easy. 

Maybe we could try to get things together ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] 9P vs. FUSE

[Skip Tavakkolian]

> did anyone do any comparisons on 9P vs. FUSE for (mostly-)local uses ? 

fuse is a clever hack for doing user space fs in unix.
look at russ' 9pfuse in p9p.

Re: [9fans] 9P vs. FUSE

[Iruata Souza]

On 8/10/07, Enrico Weigelt <weigelt@metux.de> wrote:
>
> Hi folks,
>
>
> did anyone do any comparisons on 9P vs. FUSE for (mostly-)local uses ?
>
> There're a lot of interesting FUSE-based filesystems out there,
> and programming w/ FUSE seems to be quite easy.
>
> Maybe we could try to get things together ?
>
>
> cu
> --

9p is way simpler.

Re: [9fans] 9P vs. FUSE

[Enrico Weigelt]

* Skip Tavakkolian <9nut@9netics.com> wrote:
> > did anyone do any comparisons on 9P vs. FUSE for (mostly-)local uses ? 
> 
> fuse is a clever hack for doing user space fs in unix.
> look at russ' 9pfuse in p9p.

Okay, that's an FUSE-based 9p client. 
What about FUSE-based 9p servers ?

BTW: I'm still interested in functional differences between 
FUSE and 9P. For example, does 9P support all *nix style
inode types (ie. symlinks, devices, pipes, etc) ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] 9P vs. FUSE

[erik quanstrom]

> BTW: I'm still interested in functional differences between 
> FUSE and 9P. For example, does 9P support all *nix style
> inode types (ie. symlinks, devices, pipes, etc) ?

there are no inodes in 9p.  so no.

- erik

Re: [9fans] 9P vs. FUSE

[Lucio De Re]

> there are no inodes in 9p.  so no.

I'll have private namespaces in preference over inodes any day.

++L

Re: [9fans] 9P vs. FUSE

[Enrico Weigelt]

* erik quanstrom <quanstro@quanstro.net> wrote:
> > BTW: I'm still interested in functional differences between 
> > FUSE and 9P. For example, does 9P support all *nix style
> > inode types (ie. symlinks, devices, pipes, etc) ?
> 
> there are no inodes in 9p.  so no.

Some file attributes, which can tell an special file type ?


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] 9P vs. FUSE

[erik quanstrom]

i think that you want to read

	http://www.cs.bell-labs.com/magic/man2html/5/stat

also look at the /sys/include/libc.h for the following comments

	/* bits in Qid.type */
	/* bits in Dir.mode */

- erik

Re: [9fans] 9P vs. FUSE

[Enrico Weigelt]

* erik quanstrom <quanstro@quanstro.net> wrote:
> i think that you want to read
> 
> 	http://www.cs.bell-labs.com/magic/man2html/5/stat

Thx. 
But I don't see any not whether things like symlinks are 
represented by file modes. 


cu
-- 
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] 9P vs. FUSE

[andrey mirtchovski]

> Thx.
> But I don't see any not whether things like symlinks are
> represented by file modes.

when you figure that bit out, you'll be enlightened :)

Re: [9fans] 9P vs. FUSE

[erik quanstrom]

you might check out 9p2000.u.  plan 9 doesn't do symlinks.

- erik

Re: [9fans] 9P vs. FUSE

[Eric Van Hensbergen]

On 8/10/07, Enrico Weigelt <weigelt@metux.de> wrote:
> * Skip Tavakkolian <9nut@9netics.com> wrote:
> > > did anyone do any comparisons on 9P vs. FUSE for (mostly-)local uses ?
> >
> > fuse is a clever hack for doing user space fs in unix.
> > look at russ' 9pfuse in p9p.
>
> Okay, that's an FUSE-based 9p client.
> What about FUSE-based 9p servers ?
>

I don't think FUSE-based 9p servers makes any sense.  Over
simplifying, FUSE provides a user space API into the UNIX VFS layer --
9p provides an API and Protocol into Plan 9's file server interface
(which may be local kernel, local user, or across the network).

Plan 9's file server interface is an elegant (and small) approach,
UNIX VFS is a much larger, more complicated (and many would argue more
complicated and larger than necessary) interface.  Plan 9's interface
is well suited to Plan 9.  UNIX's interface is well suited to UNIX.

Now, that being said, we have 9p under UNIX (both in p9p and v9fs, and
many other incantations), and it seems to work just fine for many
things.  We took a stab at extending 9p to match some of the UNIX
semantics (links, special files, etc.) and it was called 9p2000.u and
is implemented in the v9fs client and the spfs server code (there is
an RFC if you google for it).

However, it was decided at the last IWP9 that we had probably taken
the wrong approach with 9p2000.u and it would probably have been
better to extend 9p with new operations that had different
syntax/semantics from standard 9p as these would be easier to filter
out.  I'm currently playing with a new design in my copious spare
time.

It was further suggested by some that a better approach on Linux/UNIX
would have been to take what we know from 9p and design a protocol
specific to the VFS (similar to FUSE but capable of transparent
network, etc.)

Some of the recent v9fs effort has been looking at 9p for
paravirtualized file system access between hosts using shared memory
transports.   Much initial work has been done using 9p and 9p2000.u,
but requests for more comprehensive solutions have been requested by
customers/interested-parties with full support for Linux capabilities.
 As such, there may be a foray into providing something along the
lines of a new set of extensions supporting all Linux VFS operations
(perhaps I'll call it 9p2000.l)

However, such support is mainly necessary for folks looking to remote
access feature rich on-disk file-systems.  I believe straight 9p is
more than adequate for 99% of synthetic file systems with something as
simple as 9p2000.u giving you extra bits necessary for basic UNIX
compatibility.

               -eric

Re: [9fans] 9P vs. FUSE

[ron minnich]

On 8/10/07, Eric Van Hensbergen <ericvh@gmail.com> wrote:

>
> However, it was decided at the last IWP9 that we had probably taken
> the wrong approach with 9p2000.u and it would probably have been
> better to extend 9p with new operations that had different
> syntax/semantics from standard 9p as these would be easier to filter
> out.

And, before anyone jumps on this idea too hard, this approach works;
it's how I solved this messy problem in 1998. It was well worth trying
the .u variant, however, as you can't know some things until you try
them :-)

> It was further suggested by some that a better approach on Linux/UNIX
> would have been to take what we know from 9p and design a protocol
> specific to the VFS (similar to FUSE but capable of transparent
> network, etc.)

That's probably true. What's weird is the first one I did (for 2.0.3x)
fit into Linux VFS layer just fine. That's because the Linux VFS was
not as "mature" as it is now, and it was actually far easier in 1998
than it is now to fit a "non-standard" VFS into Linux. Back in 1998 I
had
- plan 9 style union mounts
- private name spaces that worked for all programs
- 9p mounts

and it was all pretty easy. The newer VFS layer has frozen more design
assumptions into practice, with the result that it is harder, not
easier, to put interesting file systems into Linux. It was a bit
harder for 2.2, even harder for 2.4 and, as you know, a pain in the
neck in 2.6. It's easier, I suppose, to put boring file systems in,
that act like all the other ones. There's a paper in that reality
somewhere ...

> However, such support is mainly necessary for folks looking to remote
> access feature rich on-disk file-systems.  I believe straight 9p is
> more than adequate for 99% of synthetic file systems with something as
> simple as 9p2000.u giving you extra bits necessary for basic UNIX
> compatibility.

I think that you are absolutely right. My experience certainly
supports this statement ...

It's an interesting idea to rethink v9fs with the current Linux VFS in
mind. I had not heard that one. It makes a lot of sense, however.

At the Google meeting, one discussion about putting private name
spaces in-kernel (as opposed to the current 9p-fuse which several at
Google are using) was to just have a per-user name space, mounted at
some well known place, and work with that. This was also very similar
to what I did on the original: codify the mount point to be at
/private: all users had to do a private mount (which was not a
privileged operation, hence easy) at /private; no user could see
anothers private mount; and I never had to deal with the issue of
multi-user access to single file, which has caused some work (see the
code). On the old stuff, for each private namespace file, there was
only one user, and you figured out who it was by looking in the
"inode". Going to one user per mount might also make life simpler.
That would, however, go against another idea, which was to use 9p to
serve root partitions. I'm not a big believer in network-mounted root
file systems, so am less concerned with this :-)

ron

Re: [9fans] 9P vs. FUSE

[Latchesar Ionkov]

It is not that hard to create few setuid helper programs that make
Linux support Plan9-like private namespaces. The union mount would be
tricky, is unionfs accepted in the standard kernel yet?

Thanks,
    Lucho

On 8/10/07, ron minnich <rminnich@gmail.com> wrote:
> On 8/10/07, Eric Van Hensbergen <ericvh@gmail.com> wrote:
>
> >
> > However, it was decided at the last IWP9 that we had probably taken
> > the wrong approach with 9p2000.u and it would probably have been
> > better to extend 9p with new operations that had different
> > syntax/semantics from standard 9p as these would be easier to filter
> > out.
>
> And, before anyone jumps on this idea too hard, this approach works;
> it's how I solved this messy problem in 1998. It was well worth trying
> the .u variant, however, as you can't know some things until you try
> them :-)
>
> > It was further suggested by some that a better approach on Linux/UNIX
> > would have been to take what we know from 9p and design a protocol
> > specific to the VFS (similar to FUSE but capable of transparent
> > network, etc.)
>
> That's probably true. What's weird is the first one I did (for 2.0.3x)
> fit into Linux VFS layer just fine. That's because the Linux VFS was
> not as "mature" as it is now, and it was actually far easier in 1998
> than it is now to fit a "non-standard" VFS into Linux. Back in 1998 I
> had
> - plan 9 style union mounts
> - private name spaces that worked for all programs
> - 9p mounts
>
> and it was all pretty easy. The newer VFS layer has frozen more design
> assumptions into practice, with the result that it is harder, not
> easier, to put interesting file systems into Linux. It was a bit
> harder for 2.2, even harder for 2.4 and, as you know, a pain in the
> neck in 2.6. It's easier, I suppose, to put boring file systems in,
> that act like all the other ones. There's a paper in that reality
> somewhere ...
>
> > However, such support is mainly necessary for folks looking to remote
> > access feature rich on-disk file-systems.  I believe straight 9p is
> > more than adequate for 99% of synthetic file systems with something as
> > simple as 9p2000.u giving you extra bits necessary for basic UNIX
> > compatibility.
>
> I think that you are absolutely right. My experience certainly
> supports this statement ...
>
> It's an interesting idea to rethink v9fs with the current Linux VFS in
> mind. I had not heard that one. It makes a lot of sense, however.
>
> At the Google meeting, one discussion about putting private name
> spaces in-kernel (as opposed to the current 9p-fuse which several at
> Google are using) was to just have a per-user name space, mounted at
> some well known place, and work with that. This was also very similar
> to what I did on the original: codify the mount point to be at
> /private: all users had to do a private mount (which was not a
> privileged operation, hence easy) at /private; no user could see
> anothers private mount; and I never had to deal with the issue of
> multi-user access to single file, which has caused some work (see the
> code). On the old stuff, for each private namespace file, there was
> only one user, and you figured out who it was by looking in the
> "inode". Going to one user per mount might also make life simpler.
> That would, however, go against another idea, which was to use 9p to
> serve root partitions. I'm not a big believer in network-mounted root
> file systems, so am less concerned with this :-)
>
> ron
>
>

Re: [9fans] 9P vs. FUSE

[Eric Van Hensbergen]

On 8/10/07, Latchesar Ionkov <lucho@gmx.net> wrote:
> It is not that hard to create few setuid helper programs that make
> Linux support Plan9-like private namespaces. The union mount would be
> tricky, is unionfs accepted in the standard kernel yet?
>

Its in -mm, and now there is the secondary union directory (more like
Plan 9 mounts) that is going in as well.

            -eric

Re: [9fans] 9P vs. FUSE

[ron minnich]

On 8/10/07, Latchesar Ionkov <lucho@gmx.net> wrote:
> It is not that hard to create few setuid helper programs that make
> Linux support Plan9-like private namespaces. The union mount would be
> tricky, is unionfs accepted in the standard kernel yet?
>

IIRC the unionfs that is out there is nothing like plan 9 union
mounts. Mine did the simpler Plan 9 thing. I wonder if we could just
get that in. I would have to forward-port 9 year old code. Hmm.

Deep unions with whiteouts give me the creeps.

ron

Re: [9fans] 9P vs. FUSE

[Latchesar Ionkov]

It is not only matter to forward-port it, but also get it accepted
into the kernel. I have to check what is the other union mount that
Eric mentioned.

Deep unions are not that bad as long as you don't have to write and
maintain the code :)



On 8/10/07, ron minnich <rminnich@gmail.com> wrote:
> On 8/10/07, Latchesar Ionkov <lucho@gmx.net> wrote:
> > It is not that hard to create few setuid helper programs that make
> > Linux support Plan9-like private namespaces. The union mount would be
> > tricky, is unionfs accepted in the standard kernel yet?
> >
>
> IIRC the unionfs that is out there is nothing like plan 9 union
> mounts. Mine did the simpler Plan 9 thing. I wonder if we could just
> get that in. I would have to forward-port 9 year old code. Hmm.
>
> Deep unions with whiteouts give me the creeps.
>
> ron
>
>

Re: [9fans] 9P vs. FUSE

[Eric Van Hensbergen]

On 8/10/07, Latchesar Ionkov <lucho@gmx.net> wrote:
> It is not only matter to forward-port it, but also get it accepted
> into the kernel. I have to check what is the other union mount that
> Eric mentioned.

http://lkml.org/lkml/2007/7/30/193

On a somewhat related topic, we'll probably want to enable some sort
of simple copy-on-write mode for paravirtualized file-system 9p
servers -- it strikes me that it might be better done in a single
server (cow semantics+whiteout along with UNIX file gateway service).

               -eric

Re: [9fans] 9P vs. FUSE

[Uriel]

> BTW: I'm still interested in functional differences between
> FUSE and 9P. For example, does 9P support all *nix style
> inode types (ie. symlinks, devices, pipes, etc) ?

As others have noted, no. And that is a feature, the whole point of 9P
is that all files are equal, and doing away with abominations like
symlinks, device nodes and ioctls.

The only good thing about FUSE is that it confirms that the lunix
people have not learned anything in the last thirty years.

uriel

Re: [9fans] 9P vs. FUSE

[Uriel]

> However, it was decided at the last IWP9 that we had probably taken
> the wrong approach with 9p2000.u and it would probably have been
> better to extend 9p with new operations that had different
> syntax/semantics from standard 9p as these would be easier to filter
> out.  I'm currently playing with a new design in my copious spare
> time.
>
> It was further suggested by some that a better approach on Linux/UNIX
> would have been to take what we know from 9p and design a protocol
> specific to the VFS (similar to FUSE but capable of transparent
> network, etc.)
>
> Some of the recent v9fs effort has been looking at 9p for
> paravirtualized file system access between hosts using shared memory
> transports.   Much initial work has been done using 9p and 9p2000.u,
> but requests for more comprehensive solutions have been requested by
> customers/interested-parties with full support for Linux capabilities.
>  As such, there may be a foray into providing something along the
> lines of a new set of extensions supporting all Linux VFS operations
> (perhaps I'll call it 9p2000.l)

Strange, this is not what I remember from IWP9 at all. What I remember
is that pretty much all requirements people had could be accommodated
*without* changing the existing 9p2000 protocol by using conventions
on special attach names to provide access to extra required
functionality or other such tricks (the exception was the idea of how
to group messages, which unfortunately nemo seems to have discarded
but I still think is so far the only real improvement 9p might need,
and it could be largely backwards compatible)

Of course, my memory could be wrong, but it would be really sad to see
yet another 9p variant pop up after the .u debacle when I think the
consensus was clear that 9p could handle just fine pretty much
everything everyone wanted (again with the exception of the message
grouping for high latency links).

In any case, it would be nice if people considering changes to the
protocol could be a bit more open about it so we could have some
discussion about how much sense it makes, and we could avoid a repeat
of the waste of time and effort with .u.

Best wishes

uriel

Re: [9fans] 9P vs. FUSE

[ron minnich]

On 8/10/07, Uriel <uriel99@gmail.com> wrote:
> In any case, it would be nice if people considering changes to the
> protocol could be a bit more open about it so we could have some
> discussion about how much sense it makes, and we could avoid a repeat
> of the waste of time and effort with .u.

A negative result is still useful. It makes no sense to characterize
.u this way (and I'm one of the guys who never liked it).

We learned something of value. We have the code base and experience.
That's what research is.

Openness? We were very open to people who actually contributed code.

ron

Re: [9fans] 9P vs. FUSE

[Charles Forsyth]

> is that all files are equal, and doing away with abominations like
> symlinks, device nodes and ioctls.

the 9P semantics for those things would be `wrong' (for the suggested use) anyway.
a remote 9P's device node would access the remote device, not the local one, which
is useful, but not for remotely-mounting roots.

the basic problem, which i expect will never be addressed by the linux people,
is that mknod and major and minor device numbers were of their time and now
past their prime.  (that's just one of the things that ``smells really bad''.)

Re: [9fans] 9P vs. FUSE

[erik quanstrom]

> > is that all files are equal, and doing away with abominations like
> > symlinks, device nodes and ioctls.
> 
> the 9P semantics for those things would be `wrong' (for the suggested use) anyway.
> a remote 9P's device node would access the remote device, not the local one, which
> is useful, but not for remotely-mounting roots.

ioctl allows one to pass a pointer into the kernel, doesn't it?
it's kind of hard to pass a pointer via 9p. ;-)

i believe that research unix also suffered from this problem,
but hacked around it by including 64 bytes starting with
the ptr in question.

> the basic problem, which i expect will never be addressed by the linux people,
> is that mknod and major and minor device numbers were of their time and now
> past their prime.  (that's just one of the things that ``smells really bad''.)

one of the things about plan 9 that looks better and better as you use
it is the fact that devices are fileservers, not special inodes.  this allows
one device to present many files easily, reducing the need for the things
in unix that didn't work so well.

- erik

Re: [9fans] 9P vs. FUSE

[Charles Forsyth]

>> In any case, it would be nice if people considering changes to the
>> protocol could be a bit more open about it so we could have some
>> discussion about how much sense it makes, and we could avoid a repeat
>> of the waste of time and effort with .u.

actually, i'm fairly sure i saw discussions go past on some (relevant) list

Re: [9fans] 9P vs. FUSE

[Eric Van Hensbergen]

On 8/10/07, Uriel <uriel99@gmail.com> wrote:
>
> Strange, this is not what I remember from IWP9 at all. What I remember
> is that pretty much all requirements people had could be accommodated
> *without* changing the existing 9p2000 protocol by using conventions
> on special attach names to provide access to extra required
> functionality or other such tricks (the exception was the idea of how
> to group messages, which unfortunately nemo seems to have discarded
> but I still think is so far the only real improvement 9p might need,
> and it could be largely backwards compatible)
>

(Sigh)

IIRC there were several proposed extensions such as caching, security,
(and perhaps extended attributes) that could be done under alternate
aname semantics.

However, for more complicated experiments (such as the message
groupings) would use new operations which could be easily filtered out
by servers which didn't support them instead of the mess we have with
.u and different operands for existing operations.

>
> Of course, my memory could be wrong, but it would be really sad to see
> yet another 9p variant pop up after the .u debacle when I think the
> consensus was clear that 9p could handle just fine pretty much
> everything everyone wanted (again with the exception of the message
> grouping for high latency links).
>
> In any case, it would be nice if people considering changes to the
> protocol could be a bit more open about it so we could have some
> discussion about how much sense it makes, and we could avoid a repeat
> of the waste of time and effort with .u.
>

Yes - you are right, its much better to work things out in committee
versus actually write some code to figure out how things work out in
practice.  That's just the way its always been done in Plan 9.  And so
many people had so much time wasted in the .u effort -- all those
hundreds of programmers working thousands of hours on adding .u
support, all for nothing.

Oh, wait -- there was only one other programmer working on this stuff
besides me, sorry lucho.

Vote with your code, otherwise please keep to your elite "development" lists.

         -eric

Re: [9fans] 9P vs. FUSE

[Latchesar Ionkov]

On 8/10/07, Uriel <uriel99@gmail.com> wrote:
> after the .u debacle when I think the

May be the .u debacle happened only in your head?

> In any case, it would be nice if people considering changes to the
> protocol could be a bit more open about it so we could have some
> discussion about how much sense it makes, and we could avoid a repeat
> of the waste of time and effort with .u.

I don't recall you wasting any time or effort on it.

Thanks,
    Lucho


Why? So you can hear your opin
>
> Best wishes
>
> uriel
>
>

Re: [9fans] 9P vs. FUSE

[ron minnich]

On 8/10/07, Charles Forsyth <forsyth@terzarima.net> wrote:

> actually, i'm fairly sure i saw discussions go past on some (relevant) list
>
>

yep, it was very active discussion, actually, on the v9fs list.

ron