[9fans] Distributed filesystems: Plan 9 vs. Linux

[9fans] Distributed filesystems: Plan 9 vs. Linux

[John E. Barham]

I'm working w/ a system that stores 10's of thousands of high-definition
images per project (think movies).  Storing all of the files on a single
file server is not feasible (even w/ 3 x 1.8 TB arrays per Linux file
server) and even if it were possible would be sub-optimal from a network
standpoint since literally hundreds of image processing nodes could be
hitting the server simultaneously.

Keeping track of where the files for a particular project are physically
stored is a nuisance and it can be time-consuming to track down the location
of a particular file, esp. for some of the less technically minded users
(e.g., artists).  Our solution is to develop a virtual mapping service that
maps logical URLs (e.g., /project/shot/frame010000.tiff) to physical
location (e.g., /server10/home/user/project/shot/frame010000.tiff), but this
requires adding support to client applications to work properly.  Cumbersome
but still a lot cheaper than buying high-end dedicated storage solutions,
and even then we'd run out of space.  (We considered using HTTP redirection
but couldn't see how to make this work efficiently for PUT operations.)  It
occurred to me that Plan 9 has already solved this problem by being able to
(securely) mount remote filesystems and do a union on directories.  (Correct
me if I'm wrong, but IIRC creating a file in a unioned folder would add the
file to the original folder location.)

Even something as seemingly simple as collecting stats on drive usage per
server is a pain on Linux.  For the moment we're running du over ssh using
an expect style module in Python.  Again, Plan 9 would make that trivial
either by mounting the remote file server directly, or possibly running the
script after doing a cpu connection.

Anyway, it's a truism that the more powerful hardware gets (in both CPU
power and storage capacity) the more ways we find to use it to capacity, so
mechanisms that Plan 9 provides to present a seamless view of distributed
resources are becoming more necessary, not less.

    John

P.S.  Forgive me if I'm hazy on the details of the Plan 9 commands, but the
website appears to be down at the moment...

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[andrey mirtchovski]

It's not that you can't do it in Lunix -- look at Globus' Replica
Location Service or MDS (Metasomething Discovery Service, they keep
changing the name) for just a few of the options for handling
replication, there are others specifically tailored to clusters too.
They are all crufty, don't play well with each other, require their
own clients and a pain in the proverbial to administer, but that's a
consequence of the multiple layers of abstraction they're built
upon to avoid admitting that UNIX simply lacks a decent
mechanism for importing remote resources.

Binding remote exports in a union directory is so trivial in Plan 9
that it's really a non-issue (you'll spend much more time tracking
which files are stored where, especially if they move).  The show
stoppers are the clients -- what will you be using on the client side?
Will you require users to be running Plan 9 to access the service, or
are you prepared to spend some time developing a way to bring it to
their environment (by, say, implementing a 9p client suitable for the
task, something like v9fs)...

Plan 9 lets you do something elegantly, bringing the result to the
outside world may be a bit trickier :)

my $0.02: andrey

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[boyd, rounin]

> It's not that you can't do it in Lunix -- look at Globus' Replica
> Location Service or MDS (Metasomething Discovery Service, they keep
> changing the name) ...

Meta Drongo Service

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[John E. Barham]

> It's not that you can't do it in Lunix -- look at Globus' Replica
> Location Service or MDS (Metasomething Discovery Service, they keep
> changing the name) for just a few of the options for handling
> replication, there are others specifically tailored to clusters too.

Look interesting.  It would be nice to avoid reinventing another wheel...

> They are all crufty, don't play well with each other, require their
> own clients and a pain in the proverbial to administer, but that's a
> consequence of the multiple layers of abstraction they're built
> upon to avoid admitting that UNIX simply lacks a decent
> mechanism for importing remote resources.

Yeah, the major disadvantage vis a vis Plan 9 is that they still requirement
integration w/ our client software.

> Binding remote exports in a union directory is so trivial in Plan 9
> that it's really a non-issue (you'll spend much more time tracking
> which files are stored where, especially if they move).  The show
> stoppers are the clients -- what will you be using on the client side?

Our clients operating systems are OS X, Windows and Linux.  Client-side
languages are C/C++ and Python.

> Will you require users to be running Plan 9 to access the service, or
> are you prepared to spend some time developing a way to bring it to
> their environment (by, say, implementing a 9p client suitable for the
> task, something like v9fs)...

At this point nobody is running Plan 9, and there are too many dependencies
client-side to consider P9 there.  But running P9 on the file servers and
headless image processing nodes (Mac G5s, BTW) might be worth considering at
some point.

    John

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[matt]

I have quite a bit of 9p (client & server) done in python if that interests anyone

m

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[Martin C.Atkins]

On Mon, 1 Mar 2004 17:37:18 0000 matt@proweb.co.uk wrote:
> I have quite a bit of 9p (client & server) done in python if that interests anyone
> 
> m
> 
I've got Russ' 9p Python code. It was very nice, but it would be interesting to see
some other approaches to compare...

Thanks,

Martin

-- 
Martin C. Atkins			martin@parvat.com
Parvat Infotech Private Limited		http://www.parvat.com{/,/martin}

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[Jeff Sickel]

On Feb 29, 2004, at 1:59 PM, John E. Barham wrote:
...
> (securely) mount remote filesystems and do a union on directories.
> (Correct
> me if I'm wrong, but IIRC creating a file in a unioned folder would
> add the
> file to the original folder location.)
> ...

When you can look at the web pages again:

http://plan9.bell-labs.com/magic/man2html/1/bind

If for some reason you can't go with a Plan 9 solution, you might want
to check out The Handle System (http://www.handle.net/).  Come to think
of it, adding hdl support to Plan 9 might be a nice project on it's
own.

jas

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[John E. Barham]

> If for some reason you can't go with a Plan 9 solution, you might want
> to check out The Handle System (http://www.handle.net/).  Come to think
> of it, adding hdl support to Plan 9 might be a nice project on it's
> own.

Well, at this point using Plan 9 is really not feasible since we already
have a substantial production system in place w/ dependencies that Plan 9
doesn't provide.

The Handle System looks pretty close to what we need.  My major concern
would be that it runs on Java (e.g., that "What do I do if I get a Java
out-of-memory error?" is a FAQ does not inspire confidence...)

    John

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[Jeff Sickel]

On Mar 1, 2004, at 11:14 AM, John E. Barham wrote:
>
> The Handle System looks pretty close to what we need.  My major concern
> would be that it runs on Java (e.g., that "What do I do if I get a Java
> out-of-memory error?" is a FAQ does not inspire confidence...)

Yikes, CNRI went all Java with the Handle System server software--at 
least the client libraries are still C.  I remember early versions that 
were Python based, but alas they too have been lulled into Java on the 
server.  It seems that CNRI has slowly been purging itself of all the 
Python source that had been developed there (could be one of the 
reasons Guido lef).

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[James Horey]

> occurred to me that Plan 9 has already solved this problem by being able to
> (securely) mount remote filesystems and do a union on directories.
> (Correct me if I'm wrong, but IIRC creating a file in a unioned folder
> would add the file to the original folder location.)

If I remember my reading of the documentation, I thought that when adding a
file to a unioned directory, that file is placed within the first directory
that was unioned and is writable. Otherwise the write fails completely. Am I
wrong on this or not up to date? If I am wrong, is it now possible to select
which directory you want to place your file into?

-James Horey

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[Geoff Collyer]

files are created in the first directory in the union that's creatable
(i.e.  was bound with -c).  if that attempt fails, the create fails.

Re: [9fans] Distributed filesystems: Plan 9 vs. Linux

[YAMANASHI Takeshi]

I'm thinking if this can be called a distributed filesystem?

1: import file storages from multiple servers.
2: create arena files on each imported file storage.
3: start a venti/fossil using the imported arena files.

Thus, you can create a single filesystem exploiting
multiple disk storages on multiple servers.
--