[9fans] FileServer grid

[9fans] FileServer grid

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 665 bytes --]

Hello every one...
In a context of really heavy load and high availability needs, I'm
evaluating plan 9 to implement a fileserver grid to be used by a web server
for temporary storage (session's serializations, for example).

I'd like to build a Plan 9 grid exposing a unique filesystem mounted by all
the web servers.

Each session could be accessible from any web server instantly, but what if
a fileserver in the grid break?
Is there a way to mantain such a session (actually the file storing the
session data) available by keeping it sincronized beetween the Plan 9
fileservers?


Hoping to have been clear...
Thanks for your help


Giacomo

[-- Attachment #2: Type: text/html, Size: 705 bytes --]

Re: [9fans] FileServer grid

[Roman V. Shaposhnik]

On Fri, 2009-02-06 at 23:26 +0100, Giacomo Tesio wrote:
> Hello every one...
> In a context of really heavy load and high availability needs, I'm
> evaluating plan 9 to implement a fileserver grid to be used by a web
> server for temporary storage (session's serializations, for example).

What OS do you web servers run under?

> I'd like to build a Plan 9 grid exposing a unique filesystem mounted
> by all the web servers.

Are you going to talk to this filesystem using 9P or something else?

> Each session could be accessible from any web server instantly, but
> what if a fileserver in the grid break?
> Is there a way to mantain such a session (actually the file storing
> the session data) available by keeping it sincronized beetween the
> Plan 9 fileservers?

This is unclear. Please restate.

Thanks,
Roman.

Re: [9fans] FileServer grid

[erik quanstrom]

> Hello every one...
> In a context of really heavy load and high availability needs, I'm
> evaluating plan 9 to implement a fileserver grid to be used by a web server
> for temporary storage (session's serializations, for example).
>
> I'd like to build a Plan 9 grid exposing a unique filesystem mounted by all
> the web servers.
>
> Each session could be accessible from any web server instantly, but what if
> a fileserver in the grid break?
> Is there a way to mantain such a session (actually the file storing the
> session data) available by keeping it sincronized beetween the Plan 9
> fileservers?

my 2¢.

before plan 9 got a shiny osi license, i did something like this
on linux.  since it was linux, the backend server took commands
much like a scsi device does rather than use 9p.  since there
were no reasonable threading option in those days, it worked
like a task-based operating system.  we used select to schedule
work.  all state was written to disk so that the death of the server
was not fatal.  horrible, i know.  but it worked reasonably well
without declaring war on the system we needed to use.  having
two web servers would have been easy in this setup.  having
multiple back-end servers was not possible.  but neither was ever
a performance limitation for us so it remained unexplored.
this is because all the work was pushed to a cluster of workers
fronted by the backend server.  they were always the limitation.

at some level of throughput, this server would be a limitation.
i would be suprised if the limitation would not be the speed of
your network.  since the amount of traffic this server sees should
be much less than the amount of web traffic, one machine with
a 10gbe card could move a lot of data.

- erik

Re: [9fans] FileServer grid

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 1792 bytes --]

I'd like to move our softwares to Linux + Apache (where mounting a 9p
fileserver would be easy), but actually it's a Windows + IIS.

I would write a session state service for ASP.NET connecting it in 9p (using
c# and the 9pc implementation linked by http://9p.cat-v.org/implementations...
actually the link is broken...)

If the filesystem grid work as I've understood, there would be only ONE
filesystem.

So, saving session state in the grid would make it available to all web
servers connected to the filesystem, allowing load balance and high
availability for the web servers (when one crash, the user sessions it was
handling would be available to the others web server).

But what if a node of the grid goes down? There would be a way to keep files
in it replicated in other cpu node?


Giacomo

On Fri, Feb 6, 2009 at 11:41 PM, Roman V. Shaposhnik <rvs@sun.com> wrote:

> On Fri, 2009-02-06 at 23:26 +0100, Giacomo Tesio wrote:
> > Hello every one...
> > In a context of really heavy load and high availability needs, I'm
> > evaluating plan 9 to implement a fileserver grid to be used by a web
> > server for temporary storage (session's serializations, for example).
>
> What OS do you web servers run under?
>
> > I'd like to build a Plan 9 grid exposing a unique filesystem mounted
> > by all the web servers.
>
> Are you going to talk to this filesystem using 9P or something else?
>
> > Each session could be accessible from any web server instantly, but
> > what if a fileserver in the grid break?
> > Is there a way to mantain such a session (actually the file storing
> > the session data) available by keeping it sincronized beetween the
> > Plan 9 fileservers?
>
> This is unclear. Please restate.
>
> Thanks,
> Roman.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2356 bytes --]

Re: [9fans] FileServer grid

[Uriel]

The link is *not* broken, plan9.bell-labs.com is, sadly and
unsurprisingly, broken.

uriel

On Sat, Feb 7, 2009 at 6:32 PM, Giacomo Tesio <giacomo@tesio.it> wrote:
> I'd like to move our softwares to Linux + Apache (where mounting a 9p
> fileserver would be easy), but actually it's a Windows + IIS.
>
> I would write a session state service for ASP.NET connecting it in 9p (using
> c# and the 9pc implementation linked by
> http://9p.cat-v.org/implementations... actually the link is broken...)
>
> If the filesystem grid work as I've understood, there would be only ONE
> filesystem.
>
> So, saving session state in the grid would make it available to all web
> servers connected to the filesystem, allowing load balance and high
> availability for the web servers (when one crash, the user sessions it was
> handling would be available to the others web server).
>
> But what if a node of the grid goes down? There would be a way to keep files
> in it replicated in other cpu node?
>
>
> Giacomo
>
> On Fri, Feb 6, 2009 at 11:41 PM, Roman V. Shaposhnik <rvs@sun.com> wrote:
>>
>> On Fri, 2009-02-06 at 23:26 +0100, Giacomo Tesio wrote:
>> > Hello every one...
>> > In a context of really heavy load and high availability needs, I'm
>> > evaluating plan 9 to implement a fileserver grid to be used by a web
>> > server for temporary storage (session's serializations, for example).
>>
>> What OS do you web servers run under?
>>
>> > I'd like to build a Plan 9 grid exposing a unique filesystem mounted
>> > by all the web servers.
>>
>> Are you going to talk to this filesystem using 9P or something else?
>>
>> > Each session could be accessible from any web server instantly, but
>> > what if a fileserver in the grid break?
>> > Is there a way to mantain such a session (actually the file storing
>> > the session data) available by keeping it sincronized beetween the
>> > Plan 9 fileservers?
>>
>> This is unclear. Please restate.
>>
>> Thanks,
>> Roman.
>>
>>
>
>

Re: [9fans] FileServer grid

[erik quanstrom]

> The link is *not* broken, plan9.bell-labs.com is, sadly and
> unsurprisingly, broken.

this is incorrect.  bell labs web site is working just fine.

- erik

Re: [9fans] FileServer grid

[Uriel]

It isn't http://plan9.bell-labs.com/sources/plan9/sys/src/

uriel

On Sun, Feb 8, 2009 at 12:00 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>> The link is *not* broken, plan9.bell-labs.com is, sadly and
>> unsurprisingly, broken.
>
> this is incorrect.  bell labs web site is working just fine.
>
> - erik
>
>

Re: [9fans] FileServer grid

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 2473 bytes --]

Sorry Uriel... I was meaning that I wouldn't be able to download it.


BTW my main problem is to know if, in a grid of plan 9 fileservers, there
could be any kind of replication, keeping files reachable when a node goes
down.


Giacomo

On Sat, Feb 7, 2009 at 11:57 PM, Uriel <uriel99@gmail.com> wrote:

> The link is *not* broken, plan9.bell-labs.com is, sadly and
> unsurprisingly, broken.
>
> uriel
>
> On Sat, Feb 7, 2009 at 6:32 PM, Giacomo Tesio <giacomo@tesio.it> wrote:
> > I'd like to move our softwares to Linux + Apache (where mounting a 9p
> > fileserver would be easy), but actually it's a Windows + IIS.
> >
> > I would write a session state service for ASP.NET connecting it in 9p
> (using
> > c# and the 9pc implementation linked by
> > http://9p.cat-v.org/implementations... actually the link is broken...)
> >
> > If the filesystem grid work as I've understood, there would be only ONE
> > filesystem.
> >
> > So, saving session state in the grid would make it available to all web
> > servers connected to the filesystem, allowing load balance and high
> > availability for the web servers (when one crash, the user sessions it
> was
> > handling would be available to the others web server).
> >
> > But what if a node of the grid goes down? There would be a way to keep
> files
> > in it replicated in other cpu node?
> >
> >
> > Giacomo
> >
> > On Fri, Feb 6, 2009 at 11:41 PM, Roman V. Shaposhnik <rvs@sun.com>
> wrote:
> >>
> >> On Fri, 2009-02-06 at 23:26 +0100, Giacomo Tesio wrote:
> >> > Hello every one...
> >> > In a context of really heavy load and high availability needs, I'm
> >> > evaluating plan 9 to implement a fileserver grid to be used by a web
> >> > server for temporary storage (session's serializations, for example).
> >>
> >> What OS do you web servers run under?
> >>
> >> > I'd like to build a Plan 9 grid exposing a unique filesystem mounted
> >> > by all the web servers.
> >>
> >> Are you going to talk to this filesystem using 9P or something else?
> >>
> >> > Each session could be accessible from any web server instantly, but
> >> > what if a fileserver in the grid break?
> >> > Is there a way to mantain such a session (actually the file storing
> >> > the session data) available by keeping it sincronized beetween the
> >> > Plan 9 fileservers?
> >>
> >> This is unclear. Please restate.
> >>
> >> Thanks,
> >> Roman.
> >>
> >>
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 3459 bytes --]

Re: [9fans] FileServer grid

[ron minnich]

On Sun, Feb 8, 2009 at 7:24 AM, Giacomo Tesio <giacomo@tesio.it> wrote:

> BTW my main problem is to know if, in a grid of plan 9 fileservers, there
> could be any kind of replication, keeping files reachable when a node goes
> down.


This sort of thing was done to death IIRC in the 80s. It was dropped
for a while because, at the scale of file server usage in the 90s, nfs
file servers were fine. At least from what I read today, google does
it now in GFS. There is an open source version of something that
claims to do provide it based on Hadoop
(http://hadoop.apache.org/core/) although the hadoop setups I have
seen use NFS for distributing files (!). I vaguely remember gluster
talking about failover and recovery (http://www.gluster.org/).

Have you done a literature search? I don't get the feeling that you have.

Thanks

ron

Re: [9fans] FileServer grid

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 507 bytes --]

>
> Have you done a literature search? I don't get the feeling that you have.



You are right, I searched over the net only about "Plan 9 filesystem
replication" and so...


But I was always searching for a plan 9 based solution.

I already have some alternative solution (linux based), but I would like to
use plan 9 in a prodution environment (to emancipate it to my collegue
eyes...)


>
> Thanks
>
> ron
>
>
Thanks you too!

I will get a look to your suggestions!


Giacomo

[-- Attachment #2: Type: text/html, Size: 918 bytes --]

Re: [9fans] FileServer grid

[Roman Shaposhnik]

[-- Attachment #1: Type: text/plain, Size: 1049 bytes --]

On Feb 7, 2009, at 9:32 AM, Giacomo Tesio wrote:
> I'd like to move our softwares to Linux + Apache (where mounting a
> 9p fileserver would be easy), but actually it's a Windows + IIS.
>
> I would write a session state service for ASP.NET connecting it in
> 9p (using c# and the 9pc implementation linked by http://9p.cat-v.org/implementations
> ... actually the link is broken...)

I see. Well, these are the things I have no personal experience with.
Hopefully,
somebody else on this alias can help you.

> If the filesystem grid work as I've understood, there would be only
> ONE filesystem.
>
> So, saving session state in the grid would make it available to all
> web servers connected to the filesystem, allowing load balance and
> high availability for the web servers (when one crash, the user
> sessions it was handling would be available to the others web server).

So far it seems that there are no ready-made 9P-based solutions
available for what
you have in mind. At least I don't know of any.

Thanks,
Roman.

[-- Attachment #2: Type: text/html, Size: 1413 bytes --]

Re: [9fans] FileServer grid

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 863 bytes --]

>
> So far it seems that there are no ready-made 9P-based solutions available
> for what
> you have in mind. At least I don't know of any.
>

In any solution available there are costs to pay (economical or temporary
ones).

So I could write the code missinig, if the time required to write the
filesystem sincronization and the session state handler would be a month.

For more than a month the economical cost would be (more or less) equal to
the prebuild solutions in the market.

For writing the session state handler over 9P I estimate a week...
If a sincronization system for the grid nodes (or a sintetic filesystem
providing access to the grid and replicating writes among node) would
require more than 3 weeks of man work probably I've no chance to get this
solution approved.



> Thanks,
> Roman.
>


Thanks you!

Giacomo

[-- Attachment #2: Type: text/html, Size: 1336 bytes --]

Re: [9fans] FileServer grid

[ron minnich]

On Tue, Feb 10, 2009 at 12:06 AM, Giacomo Tesio <giacomo@tesio.it> wrote:

> If a sincronization system for the grid nodes (or a sintetic filesystem
> providing access to the grid and replicating writes among node) would
> require more than 3 weeks of man work probably I've no chance to get this
> solution approved.

3 weeks? I think you have a problem here. Stop now.

This is a non trivial problem. At least from what I know.

ron

Re: [9fans] FileServer grid

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

is it too much for a syntetic filesystem 9P based? or too few?



On Tue, Feb 10, 2009 at 5:37 PM, ron minnich <rminnich@gmail.com> wrote:

> On Tue, Feb 10, 2009 at 12:06 AM, Giacomo Tesio <giacomo@tesio.it> wrote:
>
> > If a sincronization system for the grid nodes (or a sintetic filesystem
> > providing access to the grid and replicating writes among node) would
> > require more than 3 weeks of man work probably I've no chance to get this
> > solution approved.
>
> 3 weeks? I think you have a problem here. Stop now.
>
> This is a non trivial problem. At least from what I know.
>
> ron
>
>

[-- Attachment #2: Type: text/html, Size: 1022 bytes --]

Re: [9fans] FileServer grid

[Roman V. Shaposhnik]

On Tue, 2009-02-10 at 09:06 +0100, Giacomo Tesio wrote:
>         So far it seems that there are no ready-made 9P-based
>         solutions available for what
>         you have in mind. At least I don't know of any.
>
> In any solution available there are costs to pay (economical or
> temporary ones).

On the side of existing tools/solutions you may find this one
interesting: http://www.ultramonkey.org/

Thanks,
Roman.

Re: [9fans] FileServer grid

[Skip Tavakkolian]

starting with something like /sys/src/cmd/nfs.c or ramfs.c
and a setup like this:

	mount /stuff /n/node0
	import node1 /stuff /n/node1
	import node2 /stuff /n/node2

fscreate and fsopen would create or open /n/node[012]/file and fswrite
would write to /n/node[012]/file.  a good developer (not necessarily a
superstar) could write it in a week.

but this assumption -- that this would work better than if all the
nodes accessed a reliable file server -- is wrong.

> is it too much for a syntetic filesystem 9P based? or too few?
>
>
>
> On Tue, Feb 10, 2009 at 5:37 PM, ron minnich <rminnich@gmail.com> wrote:
>
>> On Tue, Feb 10, 2009 at 12:06 AM, Giacomo Tesio <giacomo@tesio.it> wrote:
>>
>> > If a sincronization system for the grid nodes (or a sintetic filesystem
>> > providing access to the grid and replicating writes among node) would
>> > require more than 3 weeks of man work probably I've no chance to get this
>> > solution approved.
>>
>> 3 weeks? I think you have a problem here. Stop now.
>>
>> This is a non trivial problem. At least from what I know.
>>
>> ron
>>
>>

Re: [9fans] FileServer grid

[Enrico Weigelt]

* Giacomo Tesio <giacomo@tesio.it> wrote:

Hi,

> I'd like to move our softwares to Linux + Apache (where mounting a 9p
> fileserver would be easy), but actually it's a Windows + IIS.

my serious condolescense ;-)

> But what if a node of the grid goes down? There would be a way to keep
> files in it replicated in other cpu node?

I'm not a Plan9 expert, but I dont know if there's already a (working)
really distributed filesystem on Plan9. You can run multiple fossil
instances from different machines in the them venti, but the venti itself
isnt distributed, neither can the fossils directly synchronize, IMHO.

Actually, I'm working on something like that:

A) an distribute version of venti (not really venti, but the same idea,
and hopefully compatible w/ it ;-o) - it not just stores blocks locally,
but can also ask its peers for blocks and push own blocks to them (there
will also be some TTL or GC mechanism to kick off stale data).

B) an distributed filesystem which stores all data in venti and just
synchronizes the head pointers (root dirs or heavily changing files)
directly with its neighbours.

(venti nodes and fileservers could also sit on different machines)

There's nothing usable yet, but perhaps you'd like to join ?


cu
--
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------