[9fans] standalone cpu server wiki

[9fans] standalone cpu server wiki

[James Horey]

Hello everybody,

I am going to attempt to install a Plan 9 network with two CPU servers, one
fileserver, and of course one terminal. I've never done this before, so I was
wondering if anybody knew if the Wiki page for installing a standalone CPU
server was up to date. Also is the Wiki page for "Installing a Plan 9 File
Server" still relevant? The Wiki paged mentioned
"that this is for a stand alone fileserver. The new (Jan 2003) Fossil
fileserver is described in setting_up_fossil". What advantages does Fossil
offer over what this page describes? Thanks for anybody that can offer
advice!

-James Horey

Re: [9fans] standalone cpu server wiki

[mirtchov]

the wiki is relatively up-to-date.  give it a shot and if you see any
problems try the 9fans archives.

cpu servers are very easy -- all they need to know is where the auth
and file servers are (auth= and fs= settings in plan9.ini).

installing a file server takes a bit more work.  in your case (when
you don't need a standalone auth/cpu/fs server) I'd say your standard
installation could very easily be turned into a file server by telling
fossil to listen to the right port and starting keyfs and listen on
service.auth from cpurc:

	auth/keyfs -wp -m /mnt/keys /adm/keys >/dev/null >[2=1]
	aux/listen -q -t /rc/bin/service.auth -d /rc/bin/service tcp

still do check the documentation on the wiki -- there are some details
like compiling an auth/cpu server kernel that you may want to get
right...

the standalone file server refers to the old Plan 9 file server
written by Ken Thompson -- it's the one described in all the papers.
fossil is a replacement which doesn't require that you dedicate an
entire machine to it (the main machine at ucalgary serves as both an
auth and file server, and allows me to login to it).  the 'kfs' file
server was used as a standalone plan9 terminal before fossil was
written.

my advice is to start by adding menuconfig to plan9.ini (see the man
page) so that you can go choose and boot an old kernel you know works
whenever there's something broken in the new one -- having a sane
/rc/bin/termrc when you're editing /rc/bin/cpurc may save you a few
hours of grief :)

andrey

> Hello everybody,
>
> I am going to attempt to install a Plan 9 network with two CPU servers, one
> fileserver, and of course one terminal. I've never done this before, so I was
> wondering if anybody knew if the Wiki page for installing a standalone CPU
> server was up to date. Also is the Wiki page for "Installing a Plan 9 File
> Server" still relevant? The Wiki paged mentioned
> "that this is for a stand alone fileserver. The new (Jan 2003) Fossil
> fileserver is described in setting_up_fossil". What advantages does Fossil
> offer over what this page describes? Thanks for anybody that can offer
> advice!
>
> -James Horey

Re: [9fans] standalone cpu server wiki

[mirtchov]

> - if fossil crashes, you could clean it up and restart it with a previous
> snapshot from venti but to do that you have to have a small kfs filesystem
> anyway. (am i right? this is something that i guessed after much pain. i
> was wondering how 9fans recover the machine from a fossil crash)

you can use a live cd with Plan 9 to boot and reinitialize fossil
too, or anything that will get you to sources.cs where you can run
the binaries...

the fact that you need another fs to boot from in order to fix fossil
has been complained about before, but nobody has mentioned yet
that (fossil or no fossil) Plan 9 gives you a much more exciting (and
novel!) way of fixing things -- next time your fossil crashes send me
an email and I'll give you an account on ucalgary.ca where you can
boot your system as a terminal and fix your fossil in about 15
minutes flat.  in fact, you can fix pretty much anything from
everywhere as long as you have a decent 9pcdisk kernel to start with.

no mucking around with floppies and installations, no recovery disks.
just a 9fat partition or a floppy that lets you hop on to any Plan 9
installation in the world (to which you have permissions to
authenticate, of course) and you have the full power of a Plan 9
machine at your disposal.

andrey

Re: [9fans] standalone cpu server wiki

[vdharani]

Hi

>> I am going to attempt to install a Plan 9 network with two CPU
>> servers, one  fileserver, and of course one terminal. I've never done
>> this before, so I was  wondering if anybody knew if the Wiki page for
>> installing a standalone CPU  server was up to date. Also is the Wiki
this Wiki document is a very useful document. i have followed it a couple
of times and done the installation. it works very well.

>> page for "Installing a Plan 9 File  Server" still relevant? The Wiki
>> paged mentioned
>> "that this is for a stand alone fileserver. The new (Jan 2003) Fossil
>> fileserver is described in setting_up_fossil". What advantages does
>> Fossil  offer over what this page describes? Thanks for anybody that
>> can offer  advice!

as andrey mentioned, fossil is supposed to be the replacement fs for kfs
but i wasnt very comfortable with fossil.

these are the problems i found about using fossil as file server:

- fossil is not robust yet. (isnt it a bit uncommon in plan 9 world?)
- when you choose fossil during installation, plan 9 prepares fossil file
server and uses it. you need to setup venti and use it as the backend
later on manually. the moment you run 'snap -a', fossil can no more work
on its own. it has flushed its data blocks to venti. so, venti is needed
from now on. you may be better off if venti is in another machine. if it
is in the same machine, you must have configured it properly so that venti
can be used at runtime. plan 9 server may not boot if the machine is
configured to get a dynamically allocated address from a pool of IP
addresses from a DHCP server, is there is any misconfiguration. localhost
may be used but one needs to explicity configure it. (cant we make it
default?)
- if fossil crashes, you could clean it up and restart it with a previous
snapshot from venti but to do that you have to have a small kfs filesystem
anyway. (am i right? this is something that i guessed after much pain. i
was wondering how 9fans recover the machine from a fossil crash)

all said and done, fossil has attrative features. though i am using kfs
now, i have reserved space for fossil in my disk and sooner or later i am
going to use fossil.

finally, my suggestion to 9fans is:

- we must remove fossil from installation option as a new user may find it
very difficult.

OR

- change installation procedure in such a way that localhost is available
by default in 9pccpuf kernel, and fossil is configured to use it by
default. also the installation procedure needs to mention somewhere that
one needs to install plan 9 with kfs again and one needs to set up venti
manually.

i only hope i am right. if i am wrong please correct me.

thanks
dharani

Re: [9fans] standalone cpu server wiki

[David Presotto]

> - fossil is not robust yet. (isnt it a bit uncommon in plan 9 world?)

I think it is now as robust as kfs, which I never considered very robust.  We're
now running everything in our lab off of fossil.  We did have to fix a spate of
problems over the last month to get it there though.  There's also a lost block
problem that we haven't found, i.e., blocks that should get reused but don't.

> - when you choose fossil during installation, plan 9 prepares fossil file
> server and uses it. you need to setup venti and use it as the backend
> later on manually. the moment you run 'snap -a', fossil can no more work
> on its own. it has flushed its data blocks to venti. so, venti is needed
> from now on. you may be better off if venti is in another machine. if it
> is in the same machine, you must have configured it properly so that venti
> can be used at runtime. plan 9 server may not boot if the machine is
> configured to get a dynamically allocated address from a pool of IP
> addresses from a DHCP server, is there is any misconfiguration. localhost
> may be used but one needs to explicity configure it. (cant we make it
> default?)

You don't need to set up venti.  What you set with fossil only is the same
as you get with kfs only; a not backed up file system.  I don't want to
make venti automatic yet because its still problematic.  I'ld rather
people have to know what's going on to set that up.

I also dislike the fact that 'snap -a' makes fossil totally dependent
on venti.  We used to have our main fossil and venti as separate machines
but this forced us to put them both on one.  Rsc, jmk, and I have talked
about putting a 'venti cache' on fossil that would keep enough state
to run standalone.  That would take a lot of thought though.

>
> - we must remove fossil from installation option as a new user may find it
> very difficult.

Actually, I'm going to take out kfs.  We don't need two in the installation
and kfs is the one that's falling behind.

>
> - change installation procedure in such a way that localhost is available
> by default in 9pccpuf kernel, and fossil is configured to use it by
> default. also the installation procedure needs to mention somewhere that
> one needs to install plan 9 with kfs again and one needs to set up venti
> manually.

I thought we'ld already done that.  Perhaps it hasn't made it into the
installation cd.  I'll fix it.

Re: [9fans] standalone cpu server wiki

[David Presotto]

> > You don't need to set up venti.  What you set with fossil only is the
> > same as you get with kfs only; a not backed up file system.  I don't
> if fossil crashes, is there a way to recover?

yes, reboot the system and then run flchk to clean up.  If you've lost
some part of the fs that you need for rebooting, you're hosed, same
as for kfs.
>
> > I also dislike the fact that 'snap -a' makes fossil totally dependent
> > on venti.  We used to have our main fossil and venti as separate
> > machines but this forced us to put them both on one.  Rsc, jmk, and I
> cant venti be in another machine? is it necessary that venti needs to run
> on the same machine as fossil? i thought it should be fine.
>

You can run them on different machines.  However, now you depend on
2 systems being up in order to boot.

Re: [9fans] standalone cpu server wiki

[David Presotto]

>
> if the installation procedure is not going to change in the near term and
> if no one else is working on these issues, i can try out a fossil based
> installation and write a document which may be useful for someone new. if
> it is acceptable, we could add it to the wiki document list. any
> suggestions?

Please do.

Re: [9fans] standalone cpu server wiki

[vdharani]

hi,
> the fact that you need another fs to boot from in order to fix fossil
> has been complained about before, but nobody has mentioned yet
> that (fossil or no fossil) Plan 9 gives you a much more exciting (and
> novel!) way of fixing things -- next time your fossil crashes send me
> an email and I'll give you an account on ucalgary.ca where you can boot
> your system as a terminal and fix your fossil in about 15
> minutes flat.  in fact, you can fix pretty much anything from
> everywhere as long as you have a decent 9pcdisk kernel to start with.
that sounds really cool!!

may be it is the documentation that kills. for an experienced 9fan, it is a
good reference material. for someone new, there are many surprises. the
document is good as individual parts, but they dont seem to fit together. i
feel something is missing. if someone who is new tries to install using the
fossil method, i am sure he will be locked up somewhere.

thanks
dharani

Re: [9fans] standalone cpu server wiki

[vdharani]

>> - fossil is not robust yet. (isnt it a bit uncommon in plan 9 world?)
>
> I think it is now as robust as kfs, which I never considered very
> robust.  We're now running everything in our lab off of fossil.  We did
> have to fix a spate of problems over the last month to get it there
> though.  There's also a lost block problem that we haven't found, i.e.,
> blocks that should get reused but don't.
>
>> - when you choose fossil during installation, plan 9 prepares fossil
>> file server and uses it. you need to setup venti and use it as the
>> backend later on manually. the moment you run 'snap -a', fossil can no
>> more work on its own. it has flushed its data blocks to venti. so,
>> venti is needed from now on. you may be better off if venti is in
>> another machine. if it is in the same machine, you must have
>> configured it properly so that venti can be used at runtime. plan 9
>> server may not boot if the machine is configured to get a dynamically
>> allocated address from a pool of IP addresses from a DHCP server, is
>> there is any misconfiguration. localhost may be used but one needs to
>> explicity configure it. (cant we make it default?)
>
> You don't need to set up venti.  What you set with fossil only is the
> same as you get with kfs only; a not backed up file system.  I don't
if fossil crashes, is there a way to recover?

> want to make venti automatic yet because its still problematic.  I'ld
> rather people have to know what's going on to set that up.
agreed.

> I also dislike the fact that 'snap -a' makes fossil totally dependent
> on venti.  We used to have our main fossil and venti as separate
> machines but this forced us to put them both on one.  Rsc, jmk, and I
cant venti be in another machine? is it necessary that venti needs to run
on the same machine as fossil? i thought it should be fine.

>> - we must remove fossil from installation option as a new user may
>> find it very difficult.
> Actually, I'm going to take out kfs.  We don't need two in the
> installation and kfs is the one that's falling behind.
in that case, i think we need to explicity specify which part of "Setting
up Fossil" is done automatically by the installation and which part needs
to be done by the user. may be a couple of lines in installation procedure
could say something like "plan 9 has a nice archiving mechanism called
venti with which fossil file system could be backed up and restored when a
problem occurs. please read about how to setup venti and set up venti
manually later."

if the installation procedure is not going to change in the near term and
if no one else is working on these issues, i can try out a fossil based
installation and write a document which may be useful for someone new. if
it is acceptable, we could add it to the wiki document list. any
suggestions?

thanks
dharani

Re: [9fans] standalone cpu server wiki

[James Horey]

> > if the installation procedure is not going to change in the near term and
> > if no one else is working on these issues, i can try out a fossil based
> > installation and write a document which may be useful for someone new. if
> > it is acceptable, we could add it to the wiki document list. any
> > suggestions?
>
> Please do.

If you do write a fossil installation page (for which I'd be very grateful),
please note that there already exists a wiki page for setting up fossil; so
perhaps you can either integrate with that page or clearly explain what the
differences between that page and yours are. Thanks!

-James Horey

Re: [9fans] standalone cpu server wiki

[okamoto]

> There's also a lost block
> problem that we haven't found, i.e., blocks that should get reused but don't.

When I removed a large file tree such as /sys/src/cmd/gs and filled it up
almost same with a small changes here and there, I got confused status of
that file tree problem.  Is this the kind you are mentioning here?

> Actually, I'm going to take out kfs.  We don't need two in the installation
> and kfs is the one that's falling behind.

I agree with you, fossil can deal with long file names!

Kenji -- be caught cold these days, it's very cold here today, too...

Re: [9fans] standalone cpu server wiki

[Jack Johnson]

On Jan 21, 2004, at 9:58 PM, mirtchov@cpsc.ucalgary.ca wrote:
> novel!) way of fixing things -- next time your fossil crashes send me
> an email and I'll give you an account on ucalgary.ca where you can
> boot your system as a terminal and fix your fossil in about 15
> minutes flat.  in fact, you can fix pretty much anything from
> everywhere as long as you have a decent 9pcdisk kernel to start with.

The last time I had terminals booting off a file server I was tempted
to try this very thing, booting across the Internet.  Unfortunately, my
current available bandwidth would not make it entertaining (though my
telco just called last night to hound me about their new DSL coverage
in my neighborhood).

What's the performance like?  Would a thin client with a small cfs
partition on, say, flash have adequate performance over, say, a 256kbps
connection?

-Jack

Re: [9fans] standalone cpu server wiki

[Jack Johnson]

On Jan 22, 2004, at 8:27 AM, David Presotto wrote:
> I also dislike the fact that 'snap -a' makes fossil totally dependent
> on venti.  We used to have our main fossil and venti as separate
> machines
> but this forced us to put them both on one.  Rsc, jmk, and I have
> talked
> about putting a 'venti cache' on fossil that would keep enough state
> to run standalone.  That would take a lot of thought though.

(Talking out my ass here, but...)

What if the first snapshot were not ephemeral, and automatically put in
a directory named recovery rather than datestamped?

-Jack

Re: [9fans] standalone cpu server wiki

[mirtchov]

> What's the performance like?  Would a thin client with a small cfs
> partition on, say, flash have adequate performance over, say, a 256kbps
> connection?

booting over a slow link daily wouldn't be such a great idea..  after
all the idea is to get enough to fix your machine.  with cfs it's
going to be a bit better for reading mail or editing code, but it's a
hassle for compiling, running latex, i/o intensive stuff.  we booted a
machine in finland from ucalgary once -- not much fun until cfs is
populated :)

my measure is this -- if vncviewer is usable over the link you can
safely run a terminal with cfs.  this makes it much better than
drawterm.

right now i am writing this at home, on a terminal+cfs running in a
vmware session booted off ucalgary over a 150KB down/60 KB up DSL
link.  the latency is pretty low at 15ms (same provider at both
places), but in my case vmware offsets it by being too slow...

others probably have more interesting stories (2ed was usable at home
over a 56K modem, they say)...

andrey

Re: [9fans] standalone cpu server wiki

[a]

back in the 3ed days (actually pre-3e, bra[sz]il days), when i first
got my terminal at home i booted it over 128k ISDN from work. booting
took forever. once up, the system was useable, although the lag was
noticeable enough to be sometimes annoying. the biggest problem was
reliability. given that the most intensive activity came durring boot
time, it would often fail durring that. we eventually went with a
local kfs for home terminals, and mounting the file server and doing
appropriate namespace tricks. the reliability was still less than
great, and failures were pretty much total ("recover" never did),
but they were infrequent, anyway. i ended up writting some simple
scripts to keep the file system in sync, before replica or tra 
existed (*much* simpler: i assumed the local stuff could be blown
away if there were differences).

i've subsequently done about the same thing over ~300k and ~1m
links. they're both quite usable, and the reliability issues
(assuming roughly constant latancy and physical-level reliability)
start to pretty much go away around 300k. i personally consider 1m
the borderline of what's comfortable.
ア