[9fans] Easiest way to make a filesystem

[9fans] Easiest way to make a filesystem

[Tom Lieber]

I'd like to make a few simple filesystems for personal use to do
things such as combine files or translate between different formats --
really simple stuff, on par with the filesystems described in the
"laying namespaces" paper from IWP9.

Is the easiest way to make a filesystem to make one in C in the way
described in Francisco's book? Are there any wrapping libraries for
the simplest filesystems? Or filesystems like these to base my work
on?

shifs (Uriel) seemed encouraging, as did tmfs (Noah Evans), though I
can't find source, nor determine the viability of either without it. I
think they are for Inferno.

I just obtained trfs but have not yet given it a thorough look. It's
intimidating that it is 400 lines for character replacement, since its
purpose is to pass every request nearly without modification to the
underlying fs.

-- 
Tom Lieber
http://AllTom.com/

Re: [9fans] Easiest way to make a filesystem

[Christopher Nielsen]

have a look at /sys/lib/lib9p/ramfs.c as a guide for writing one in C.
in the past, i have found it useful and educational.

On Jan 14, 2008 1:11 AM, Tom Lieber <alltom@gmail.com> wrote:
> I'd like to make a few simple filesystems for personal use to do
> things such as combine files or translate between different formats --
> really simple stuff, on par with the filesystems described in the
> "laying namespaces" paper from IWP9.
>
> Is the easiest way to make a filesystem to make one in C in the way
> described in Francisco's book? Are there any wrapping libraries for
> the simplest filesystems? Or filesystems like these to base my work
> on?
>
> shifs (Uriel) seemed encouraging, as did tmfs (Noah Evans), though I
> can't find source, nor determine the viability of either without it. I
> think they are for Inferno.
>
> I just obtained trfs but have not yet given it a thorough look. It's
> intimidating that it is 400 lines for character replacement, since its
> purpose is to pass every request nearly without modification to the
> underlying fs.
>
> --
> Tom Lieber
> http://AllTom.com/
>



-- 
Christopher Nielsen
"They who can give up essential liberty for temporary
safety, deserve neither liberty nor safety." --Benjamin Franklin

Re: [9fans] Easiest way to make a filesystem

[Eric Van Hensbergen]

On Jan 14, 2008 3:11 AM, Tom Lieber <alltom@gmail.com> wrote:
>
> Is the easiest way to make a filesystem to make one in C in the way
> described in Francisco's book? Are there any wrapping libraries for
> the simplest filesystems? Or filesystems like these to base my work
> on?
>

Take a look at the man page for p9file(2) - its a starting point for
writing simple file systems.  I don't think there is something which
explicitly helps with stackable/layered file systems -- might be nice
if you created one out of your efforts ;)

       -eric

Re: [9fans] Easiest way to make a filesystem

[roger peppe]

i agree that it's too hard to create servers for simple filesystems.

i've been wondering on and off for years what a "low-bar-to-entry" filesystem
creating library might look like, and my current thoughts go something like:

- it's the generality of the problem that makes it hard. 9p allows a great
deal of freedom in implementing semantics. there's not one design that
fits all patterns.

- that said, if we restrict the problem, we can make things simpler.
inferno's file2chan is an extreme example. 9pfile is an example at the
other end of the spectrum. something that deals with stackable filesystems
would be another example - but how much generality do you go for?
change contents only? change names? add new hierarchies?
change blocking behaviour?

- it seems to me that most of the time we want "natural" (i.e.
standard filesystem) semantics
everywhere we don't specify something different. this can be looked at a kind
of inheritance, the difficulty being that the inheritance cuts across many
possible aspects of the implementation - fids, qids, tags - all of which are
expressed in very different ways in code.

- perhaps there's room for a new "little language" here; along
the lines of yacc, maybe. but, how would we *want* to
express the specification for a file server?

Re: [9fans] Easiest way to make a filesystem

[Tom Lieber]

On Jan 14, 2008 3:10 PM, roger peppe <rogpeppe@gmail.com> wrote:
> i agree that it's too hard to create servers for simple filesystems.
>
> i've been wondering on and off for years what a "low-bar-to-entry" filesystem
> creating library might look like, and my current thoughts go something like:
>
> - it's the generality of the problem that makes it hard. 9p allows a great
> deal of freedom in implementing semantics. there's not one design that
> fits all patterns.
>
> - that said, if we restrict the problem, we can make things simpler.
> inferno's file2chan is an extreme example. 9pfile is an example at the
> other end of the spectrum. something that deals with stackable filesystems
> would be another example - but how much generality do you go for?
> change contents only? change names? add new hierarchies?
> change blocking behaviour?
>
> - it seems to me that most of the time we want "natural" (i.e.
> standard filesystem) semantics
> everywhere we don't specify something different. this can be looked at a kind
> of inheritance, the difficulty being that the inheritance cuts across many
> possible aspects of the implementation - fids, qids, tags - all of which are
> expressed in very different ways in code.
>
> - perhaps there's room for a new "little language" here; along
> the lines of yacc, maybe. but, how would we *want* to
> express the specification for a file server?

It seems that all that really matters in a read-only filesystem is
which files are available, and what you get when you read them. I
imagine writing a filesystem in rc with two functions called by the 9p
wrapper code: one for ls, and one for reading the entire file.

A completely pass-through filesystem would return `{ ls -l
$realdir/$path } for directory listings and `{ cat $realdir/$path }
for read requests. Others would pipe file data through other programs,
or return its own list of synthetic files instead of ls output. It
would all be very inefficient and hackish and really easy to churn
out.

Writing to files? I'll think about that later.

'Course this is all talk by a guy that hasn't even finished the 9p
chapter, and I know how annoying idle chat is... I'll get back to
writing code after a few more exams.

-- 
Tom Lieber
http://AllTom.com/

Re: [9fans] Easiest way to make a filesystem

[hiro]

Has anybody tried a fsfs yet?

--
hiro

Re: [9fans] Easiest way to make a filesystem

[Gary Wright]

On Jan 14, 2008, at 3:10 PM, roger peppe wrote:
> i agree that it's too hard to create servers for simple filesystems.
>
> i've been wondering on and off for years what a "low-bar-to-entry"  
> filesystem
> creating library might look like, and my current thoughts go  
> something like:

I've been playing around with my own implementation of 9P in Ruby in  
an effort to explore this problem space and to better understand 9P.   
I don't know how many people on this list are familiar with ruby but  
it is pretty readable so I'll throw out some examples of my code and  
see what you think.  The example below does everything locally, but  
I've got facilities for connecting and serving via TCP and the 9P- 
wire protocol.

I'd appreciate comments/feedback.

class Example < Device
   class Root < Directory
     root               :mode => 0555
     child 'readonly',  :mode => 0600
     child 'text',      :mode => 0600, :contents => 'some text to  
start with'
     child 'subdir/'    :mode => 0700 do
       child 'subfile'  :mode => 0555
     end
     child 'bob'        :mode => 0600, :uid => 'bob'
     child 'all'        :mode => 0444

     # Lets define a Pipe resource
     class Pipe < Resource
       # incoming write requests get forwarded to aux object
       def write(fid, offset, data)
         aux.write(data)
       end

       # incoming read requests get forwarded to aux object
       def read(fid, offset, count)
         aux.read(count)
       end
     end

     # Everytime the Device gets instantiated construct
     # a unix domain socketpair, attach the ends to two
     # new Pipe resources and then put the Pipe resources
     # in the device namespace.
     child 'pipe/'      :mode => 0700 do
       data, data1 = *UNIXSocket.socketpair
       child 'data', Pipe, :mode => 0600, :aux => data
       child 'data1', Pipe, :mode => 0600, :aux => data1
     end
   end
end

# Everytime Example is instantiated we'll get a hiearchy like:
#  /readonly
#  /text
#  /subdir
#  /subdir/subfile
#  /bob
#  /all
#  /pipe
#  /pipe/data         data/data1 connected via Unix pipe
#  /pipe/data1

# Access the device via 9P API
device = Example.new
device.version                  # Tversion
a = device.auth                 # Tauth
r = device.attach(a)            # Tattach
data = r.walk('pipe', 'data').open(ORDWR)
data.write('some text')
data1 = r.walk('pipe', 'data1').open
puts data1.read                 # read until EOF (0 byte read)


# Now construct a Plan 9-like namespace for access

ns = NS.attach(Device.new)      # start with memory fs device mounted  
at '/'

ns.mkdir '/d1'
ns.mkdir '/d2'
ns['/d1'].mount Example.new     # instantiate Example and mount
ns['/d2'].mount Example.new     # again but on a different directory

d = ns['/d1/pipe/data'].open(ORDWR)
d.write('some txt')
puts ns['/d1/pipe/data1'].open.read    # read from the other end

Re: [9fans] Easiest way to make a filesystem

[roger peppe]

On Jan 14, 2008 9:00 PM, Gary Wright <gwtmp01@mac.com> wrote:
> I've been playing around with my own implementation of 9P in Ruby in
> an effort to explore this problem space and to better understand 9P.
[...]
> I'd appreciate comments/feedback.

how do you deal with filesystems where the name space is dynamically
constructed? for instance, how would you go about specifying the #I
(/net) namespace
with your system?

Re: [9fans] Easiest way to make a filesystem

[Gary Wright]

On Jan 14, 2008, at 4:10 PM, roger peppe wrote:

> On Jan 14, 2008 9:00 PM, Gary Wright <gwtmp01@mac.com> wrote:
>> I've been playing around with my own implementation of 9P in Ruby in
>> an effort to explore this problem space and to better understand 9P.
> [...]
>> I'd appreciate comments/feedback.
>
> how do you deal with filesystems where the name space is dynamically
> constructed? for instance, how would you go about specifying the #I
> (/net) namespace
> with your system?


The filesystem is just a data structure.  It can be manipulated
from the 9P side via create/remove (subject to all the regular
access rules) or it can be manipulated on the internal side:

# assume that 'd' is a reference to an instance of Directory

d['newfile'] = Resource.new

The default behavior of a resource is to just map file I/O to
an in memory string but you can define new resources that
intercept and interpret 9P ops at will:

class TimeOfDay < Resource
   def read(fid, offset, count)
     Time.now.to_s[0,count]
   end
   undef_method write
end

d['time'] = TimeOfDay.new

now every read attempt on 'time' will return  the time of day
and write requests will generate an error.


Gary Wright

Re: [9fans] Easiest way to make a filesystem

[Roman Shaposhnik]

On Mon, 2008-01-14 at 20:10 +0000, roger peppe wrote:
> i agree that it's too hard to create servers for simple filesystems.
> 
> i've been wondering on and off for years what a "low-bar-to-entry" filesystem
> creating library might look like, and my current thoughts go something like:

  I've been thinking about the same thing for quite some time now (and
don't you suspect that I have any real suggestions here, more like a
bunch of pet peeves). I guess my personal feeling of cognitive
dissonance comes from the fact that I can do pretty much anything in
Plan9 in a more or less simple shell script, but I can't easily "share"
the service of the shell scripts via namespace abstraction.

Thanks,
Roman.

Re: [9fans] Easiest way to make a filesystem

[roger peppe]

On Jan 14, 2008 9:29 PM, Roman Shaposhnik <rvs@sun.com> wrote:
>   I've been thinking about the same thing for quite some time now (and
> don't you suspect that I have any real suggestions here, more like a
> bunch of pet peeves). I guess my personal feeling of cognitive
> dissonance comes from the fact that I can do pretty much anything in
> Plan9 in a more or less simple shell script, but I can't easily "share"
> the service of the shell scripts via namespace abstraction.

in fact, in inferno at least, one can do a reasonable amount
with sh-file2chan(1), memfs(1), bind(1) and export(1). it's good for
putting things together on an ad hoc basis, at any rate.

for a plan 9 example, pipefile(1) is a tiny program that is constructed
similarly - the bugs section says it "should be rewritten to be a
user-level file system".
the problem is that if that was done, it would be probably three times the size.
there's such a step change from the small (but useful) things that can
be thrown together out of existing parts to writing a full-blown file server.

it's probably a pipe dream, but every time i launch into the
grunge work involved in writing a proper file server, i find myself
wishing there was something
a little more straightforward.