[9fans] file server?

[9fans] file server?

[Benjamin Huntsman]

Since the how-to's are being discussed recently, I thought it would be a good time to ask:

Once, it used to be the "standard" configuration to have one machine as a CPU/auth server, one machine as a file server, and one machine as a "terminal", for a total of three systems, if one had the available hardware.

What's the "recommended" setup now?  Are most people using a combined cpu/auth/file server running Fossil+Venti, or is the recommendation to use a seperate fossil+venti server dedicated to file serving, and another to serve as CPU/auth?

I currently have a cpu/auth server in a vm that I drawterm to and store most of my Plan 9 stuff on, but since I have an opportunity to move some things around, am wondering if this is the best setup.

Thanks to all in advance!

-Ben

Re: [9fans] file server?

[andrey mirtchovski]

I drawterm all the time. Lately I have started using 9vx for quick
hacks and then cpu from it to wherever I want to go. It has its quirks
(not as stable as drawterm) but I'm not complaining.

With drawterm being so solid and well integrated with X11/OSX I
haven't had the need to dedicate a terminal to Plan 9 since 2005. If I
need to spend a relaxing day at work I simply fire dt full-screen and
look at the colors :)

So, I guess that means venti+fossil+cpu on one headless machine in
some forgotten corner of the datacentre.

andrey

Re: [9fans] file server?

[erik quanstrom]

> So, I guess that means venti+fossil+cpu on one headless machine in
> some forgotten corner of the datacentre.

regardless of one's terminal accomidations, i still think it makes
a lot of sense to have a stand-alone fileserver.  it really does stink
if your fs goes down for no reason at all.  this is especially true if
you're doing a lot of experimenting or don't have a proper terminal.

in fact, i think it makes sense to devote two machines to one's
fileserver.  one diskless fileserver and an coraid storage appliance. :-)

shameless, aren't i?

- erik

Re: [9fans] file server?

[John Floren]

On Thu, Aug 13, 2009 at 6:01 PM, erik quanstrom<quanstro@quanstro.net> wrote:
>> So, I guess that means venti+fossil+cpu on one headless machine in
>> some forgotten corner of the datacentre.
>
> regardless of one's terminal accomidations, i still think it makes
> a lot of sense to have a stand-alone fileserver.  it really does stink
> if your fs goes down for no reason at all.  this is especially true if
> you're doing a lot of experimenting or don't have a proper terminal.
>
> in fact, i think it makes sense to devote two machines to one's
> fileserver.  one diskless fileserver and an coraid storage appliance. :-)
>
> shameless, aren't i?
>
> - erik
>
>

This is what we do at Sandia. We have one machine which serves
cpu/auth/file, but the actual Venti disks are in a Coraid connected
via GigE. The fossil disk is in the server, but if it dies we can just
build a new one.

John
-- 
"Object-oriented design is the roman numerals of computing" -- Rob Pike

Re: [9fans] file server?

[erik quanstrom]

> This is what we do at Sandia. We have one machine which serves
> cpu/auth/file, but the actual Venti disks are in a Coraid connected
> via GigE. The fossil disk is in the server, but if it dies we can just
> build a new one.

coraid's configuration using ken's fs is outlined here
	http://www.quanstro.net/plan9/disklessfs.pdf
the one change since the paper has been that the wireless connection
has been replaced by 100Mbit fiber.

my own setup is here
	http://www.quanstro.net/plan9/fs.html

the advantage to such a setup is than when disks do stupid
things, you can take care of the problem without disturbing
the fileserver.

- erik

Re: [9fans] file server?

[Lyndon Nerenberg]

> This is what we do at Sandia. We have one machine which serves
> cpu/auth/file, but the actual Venti disks are in a Coraid connected
> via GigE. The fossil disk is in the server, but if it dies we can just
> build a new one.

Which reminds me of an often overlooked but important point:

   Save your fossil vac scores on another machine!

Without them, your seperate venti server is JBOD :-P Well, not quite. You
can eventually find the right vac score, but you have to manually mount
each and every score in the venti until you find the right one. See
/sys/src/cmd/venti/words/dumpvacroots. You could probably semi-automate
the process by writing a script that mounted each of the scores in turn,
checking the mtime of something like /sys/log/timesync in each, and
sorting the vac scores accordingly.

On my setup I aux/clog the fileserver console to a u9fs mounted directory
on a UNIX server. You could also cobble something up that scans the fossil
console for vac scores and emails them to an offsite address.

--lyndon

Re: [9fans] file server?

[Robert Raschke]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On Fri, Aug 14, 2009 at 7:14 AM, Lyndon Nerenberg <lyndon@orthanc.ca> wrote:

> This is what we do at Sandia. We have one machine which serves
>> cpu/auth/file, but the actual Venti disks are in a Coraid connected
>> via GigE. The fossil disk is in the server, but if it dies we can just
>> build a new one.
>>
>
> Which reminds me of an often overlooked but important point:
>
>  Save your fossil vac scores on another machine!
>

I have a nightly cron job that sends the latest output of the fossil console
as an email.

In cpurc of the fileserver I setup a

aux/clog /srv/fscmd /sys/log/fossil &

and my nightly batch just does this:

who = (myself fossil@some.other.place)
today = `{date}
tail -20 /sys/log/fossil |mail -s 'Fossil output of ' ^ $"today $who

The last 20 lines is probably overkill, but who cares.

The venti archive starts at 2AM, and my cron job is at 4AM. So far, I've not
yet had an archive take longer than 2 hours. But that's partly due to
triggering one explicitly after a pull that's just replaced all my
executables ;-)

Robby

[-- Attachment #2: Type: text/html, Size: 1601 bytes --]

Re: [9fans] file server?

[erik quanstrom]

> The venti archive starts at 2AM, and my cron job is at 4AM. So far, I've not
> yet had an archive take longer than 2 hours. But that's partly due to
> triggering one explicitly after a pull that's just replaced all my
> executables ;-)

that's surprising to me that it would take that long.
is that typical?

just on the back of the proverbial envelope ...
a completely seek-bound disk with 8k blocks
gets ~3mb/s even when diabolicly seek bound,
lets take something way more pessimistic
like 0.5mb/s for 1 hr = 1800mb.

180mb dumps would still be large, so you
must be getting on the order of 50kb/s
dumps?

i'm not a venti expert, but i find it hard to believe
that venti would be that slow normally.  do you
need to resize memory or your bloom filter?

hope my display of ignorance here doesn't
seem like criticism.

- erik

Re: [9fans] file server?

[Robert Raschke]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

On Fri, Aug 14, 2009 at 1:07 PM, erik quanstrom <quanstro@quanstro.net>wrote:

> > The venti archive starts at 2AM, and my cron job is at 4AM. So far, I've
> not
> > yet had an archive take longer than 2 hours. But that's partly due to
> > triggering one explicitly after a pull that's just replaced all my
> > executables ;-)
>
> that's surprising to me that it would take that long.
> is that typical?
>
> just on the back of the proverbial envelope ...
> a completely seek-bound disk with 8k blocks
> gets ~3mb/s even when diabolicly seek bound,
> lets take something way more pessimistic
> like 0.5mb/s for 1 hr = 1800mb.
>
> 180mb dumps would still be large, so you
> must be getting on the order of 50kb/s
> dumps?
>
> i'm not a venti expert, but i find it hard to believe
> that venti would be that slow normally.  do you
> need to resize memory or your bloom filter?
>
> hope my display of ignorance here doesn't
> seem like criticism.
>
> - erik
>
>
I am running an old (pre bloom) venti on a 500MHz VIA with some crappy
no-name 30GB PATA drive. But, yes, I would be surprised if an archival dump
took longer than a few minutes (from looking at my logs anything from 45sec
to 4.5min is typical, but I haven't looked at the amounts of data).

Saying that, those pulls from a while ago, where pretty much the whole file
tree got replaced did lead to a dump taking almost an hour.

Robby

[-- Attachment #2: Type: text/html, Size: 1851 bytes --]

Re: [9fans] file server?

[erik quanstrom]

> Without them, your seperate venti server is JBOD :-P Well, not quite. You
> can eventually find the right vac score, but you have to manually mount
> each and every score in the venti until you find the right one. See
> /sys/src/cmd/venti/words/dumpvacroots. You could probably semi-automate
> the process by writing a script that mounted each of the scores in turn,
> checking the mtime of something like /sys/log/timesync in each, and
> sorting the vac scores accordingly.

any reason fossil can't haul its own water?

- erik

Re: [9fans] file server?

[Adrian Tritschler]

2009/8/14 Lyndon Nerenberg <lyndon@orthanc.ca>:
>> This is what we do at Sandia. We have one machine which serves
>> cpu/auth/file, but the actual Venti disks are in a Coraid connected
>> via GigE. The fossil disk is in the server, but if it dies we can just
>> build a new one.
>
> Which reminds me of an often overlooked but important point:
>
>  Save your fossil vac scores on another machine!

Amen, a dozen times!  Having lost the lot once or twice due to
stupidity or hardware failures I've now got this under control, still
not entirely happy with my venti arena archiving and seeking
inspiration there.

I'll add the description of my trivial home system to the mix. A
ubuntu linux PC running p9p's venti, a plan9 cpu/fossil-fileserver
running under QEMU and drawterm.  A few times I've tried running 9vx
but for unknown reasons I can crash it as soon as I look at it
sideways - I suspect either operator error or oddities of my setup.
All running on a P4 shuttle system tucked into the corner of my desk.

> Without them, your seperate venti server is JBOD :-P Well, not quite. You
> can eventually find the right vac score, but you have to manually mount each
> and every score in the venti until you find the right one. See
> /sys/src/cmd/venti/words/dumpvacroots. You could probably semi-automate the
> process by writing a script that mounted each of the scores in turn,
> checking the mtime of something like /sys/log/timesync in each, and sorting
> the vac scores accordingly.
>
> On my setup I aux/clog the fileserver console to a u9fs mounted directory on
> a UNIX server. You could also cobble something up that scans the fossil
> console for vac scores and emails them to an offsite address.

For me, cron runs fossil/last and mails it to gmail

> --lyndon
  Adrain

-- 
Adrian

Re: [9fans] file server?

[Lyndon Nerenberg]

> regardless of one's terminal accomidations, i still think it makes
> a lot of sense to have a stand-alone fileserver.  it really does stink
> if your fs goes down for no reason at all.  this is especially true if
> you're doing a lot of experimenting or don't have a proper terminal.

Amen!  Three times now I've lost everything due to a combined
CPU/fileserver crashing. The joy of a diskless CPU server is it cannot
corrupt your filesystem no matter what manner of stupid things you do.

My current fileserver runs fossil, venti, keyfs, secstored, dhcpd, and
tftpd. I.e. only the bare minimum necessary to provide fileservice and
bootstrap the diskless machines. (I will soon be moving the venti to a
different machine to eliminate the current single point of disk failure.)
It doesn't even have a keyboard hooked up. And it's rock solid. Real work
is done on the diskless CPU server. Currently I drawterm to the CPU box,
but that's because the terminal machine is in pieces waiting for some
hardware upgrades.

--lyndon

Re: [9fans] file server?

[Federico G. Benavento]

no that data center here, but hopefully at a corner of the room

On Thu, Aug 13, 2009 at 9:55 PM, andrey
mirtchovski<mirtchovski@gmail.com> wrote:
> I drawterm all the time. Lately I have started using 9vx for quick
> hacks and then cpu from it to wherever I want to go. It has its quirks
> (not as stable as drawterm) but I'm not complaining.
>
> With drawterm being so solid and well integrated with X11/OSX I
> haven't had the need to dedicate a terminal to Plan 9 since 2005. If I
> need to spend a relaxing day at work I simply fire dt full-screen and
> look at the colors :)
>
> So, I guess that means venti+fossil+cpu on one headless machine in
> some forgotten corner of the datacentre.
>
> andrey
>
>



--
Federico G. Benavento

Re: [9fans] file server?

[blstuart]

> Once, it used to be the "standard" configuration to have one machine as a CPU/auth server, one machine as a file server, and one machine as a "terminal", for a total of three systems, if one had the available hardware.

The power in that model comes primarily when you have
a number of terminals being supported by the file and cpu/auth
servers.  Throwing bigger disks on the file server gives everyone
more space.  Upgrading the big honkin' cpu server gives everyone
a speed boost.  Of course, these days, a $400 laptop is more
honkin' (in many ways) than big systems of just a few years
ago.  Still the separation of functionality does have advantages.
But I digress...

> What's the "recommended" setup now?  Are most people using a combined cpu/auth/file server running Fossil+Venti, or is the recommendation to use a seperate fossil+venti server dedicated to file serving, and another to serve as CPU/auth?

Currently, I'm running a combined file/cpu/auth server, and
I run 9vx in FreeBSD as a terminal connecting to my server.
While not mentioned often, 9vx has a -b option that allows
you to point it to a file server the same way a terminal does.
I can also cpu into the cpu server just like with a real terminal.
And when I'm away from home with my laptop, I can still
run 9vx with a local file system.

As soon as my supply of round tuits is replinished, I intend
to put a CPU server in place.

BLS