Re: [9fans] ceph

[ron minnich <rminnich@gmail.com>]

On Fri, Jul 31, 2009 at 10:53 PM, Roman Shaposhnik<rvs@sun.com> wrote:

> What are your clients running?

Linux

> What are their requirements as
> far as POSIX is concerned?

10,000 machines, working on a single app, must have access to a common
file store with full posix semantics and it all has to work like it
were one machine (their desktop, of course).

This gets messy. It turns into an exercise of attempting to manage a
competing set of race conditions. It's like tuning
a multi-carburated enging from years gone by, assuming we ever had an
engine with 10,000 cylinders.

> How much storage are talking about?
In  round numbers, for the small clusters, usually a couple hundred T.
For anyhing else, more.

>
> I'd be interested in discussing some aspects of what you're trying to
> accomplish with 9P for the HPC guys.

The request: for each of the (lots of) compute nodes, have them mount
over 9p to, say 100x fewer io nodes, each of those to run lustre.
Which tells you right away that our original dreams for lustre did not
quite work out.

In all honesty, however, the 20K node Jaguar machine at ORNL claims to
run lustre and have it all "just work". I know as many people who have
de-installed lustre as use it, however.

ron