[9fans] Help with two small shared file servers

[9fans] Help with two small shared file servers

[Aram Hăvărneanu]

Hello,

I'm looking for advice on how to build a small network of two file
servers. I'm hoping most servers to be Plan9, clients are Windows and
Mac OS X.

I have 2 houses separated by about 40ms of network latency. I want to
set some servers in each location and have all data accessible from
anywhere. I'll have about 2TB of data at each location, one location
will probably scale up.

What's the best option for RAID in Plan9? I understand I can use
either Ken's fileserver or the Plan9 '#k' device. Can anyone shed some
light on why I might want one and not the other? Are there any other
options?

Do you recommend fossil+venti or kfs? I've seen that a lot of people prefer kfs.

Are there any options for block level encryption on Plan9?

Is 9p suitable for this? How will the 40ms latency affect 9p
operation? (I have 100Mbit). I assume there are no good 9p drivers for
Windows so I can access my data directly. How good are the various
Plan9 SMB servers? (On the long term I might write a 9p filesystem
driver for Windows, I've been a Windows filesystem developer in a past
life). Is it better if I use the Plan9 boxen only for AoE with a Linux
on top that serves SMB?

I'm hoping to use some laptops for this. I have a lot of unused
ThinkPads that can run Plan9 around here. Are there some kind of
external enclosures that can hold a reasonable number of drives and
connect via eSATA or something? Are they fast enough?

On an unrelated note, what do you recommend for VPN. In Plan9 is very
easy as I can just import a namespace, but I have Windows and Mac
clients. Are there any Plan9 options? If not, what else would you
recommend?

Right now (only one location) I am using a Solaris server with ZFS
that serves SMB and iSCSI. It works great, there's nothing wrong with
it, but I would like a Plan9 solution. Or do you think iSCSI would
perform better with this latency? (I can't use AoE as AoE is not
routable).

Any tips are welcomed :-),

Thanks,

-- 
Aram Hăvărneanu

Re: [9fans] Help with two small shared file servers

[Anthony Sorace]

[-- Attachment #1: Type: text/plain, Size: 5019 bytes --]

On Aug 17, 2011, at 6:09 AM, Aram Hăvărneanu wrote:

> What's the best option for RAID in Plan9? I understand I can use
> either Ken's fileserver or the Plan9 '#k' device. 

note that neither of these are RAID in the way most people expect. failure
notification, in particular, can be lacking, and they're more restricted in what
they try to do that many RAID systems. that said, I've used #k quite happily for
years with no issues (and periodic correctness checks).

i'm not sure what support, if any, we have for hardware RAID controllers other
than ones which present a single logical disk. those, though, should be an
option if you're concerned.

> Can anyone shed some light on why I might want one and not the other?
> Are there any other options?

ken's fs is a kernel, and essentially gives you a 9p-accessible file storage
appliance. it takes another box to be able to run it. #k is part of the standard
cpu server kernel, and is typically used in conjunction with fossil and/or
venti. really, you usually want to decide between Ken's fs and fossil (with or
without venti). the basics:

Ken's FS used to be the default file server in Plan 9. It's a kernel, and takes over
a box. It is only directly accessible via 9p and il, which pretty much means via
Plan 9 (cpu servers can share the resources as they like, of course). It does
automatic daily archives of the whole system. It does limited de-duplicating.
It is very fast and extremely stable.

fossil is a newer file server, designed for use with venti (but usable on its own,
as well). it can be configured to do daily archives and/or periodic ephemeral
snapshots (you pick how often and how long they last). It can be used stand-
alone (can't do archives that way, but can still do snapshots) or with venti, in
which case you get venti's block-level deduplication (and direct access to
block-level storage). fossil performs reasonably and is relatively stable, but
less so than ken's fs (and, personally, seems more sensitive to unexpected
shutdown). venti is very stable. typically, if fossil goes belly-up, you can
reconstruct it from venti (in every case i've seen).

personally, if you (a) have the extra box to spare, (b) can put in a little extra
work up front, (c) don't care about direct access to block-level storage, and
(d) can live without ephemeral snapshots, i'd suggest ken's fs; otherwise, i'd
suggest fossil backed by venti.

(note that (d) above is slightly tricky, as the ephemeral snapshots in fossil
seem to trigger a bug that causes fossil to hang periodically; most reports
have that happening every ~2 months or so)

this gets asked often, and is worth a wiki page.

> Do you recommend fossil+venti or kfs? I've seen that a lot of people prefer kfs.

note that "kfs", confusingly, is not "ken's fs". that's sometimes (more confusingly)
informally called "kenfs". kfs is a quite old offshoot that used to be the standard
install on things like laptops or other stand-alone systems before fossil came
along.

of kenfs, kfs, and fossil (with or without venti), kfs is your most "traditional" file
system. no snapshots or archives, no connection to venti or similar, no block or
file level de-duplicating. it still has a place, but doesn't sound like what you want.

> Are there any options for block level encryption on Plan9?

there have been a handful of such experiments, but i'm pretty sure there's nothing
in the stock distribution. check the 9fans archives for discussion of the various
experiments. i'm not sure if any were quite "there" yet.

> Is 9p suitable for this? How will the 40ms latency affect 9p operation?

i expect that'll be well low enough for most uses.

> I assume there are no good 9p drivers for Windows so I can
> access my data directly.

there was at least one good one, but (last i checked) it's not generally available
(search the archives for "rangboom" for more info).

> How good are the various
> Plan9 SMB servers? (On the long term I might write a 9p filesystem
> driver for Windows, I've been a Windows filesystem developer in a past
> life). Is it better if I use the Plan9 boxen only for AoE with a Linux
> on top that serves SMB?

i've used the plan9 smb stuff only lightly, but it's worked well for me. there are
folks actively paying attention to it and interested in having it work well, which
is always good. he linux stuff will doubtless be more well-tested, but then you
(a) have an extra layer of stuff to set up and configure, and (b) have linux.

i'd give the plan9 smb stuff a shot and see how it works for you.

> Are there some kind of external enclosures that can hold a reasonable
> number of drives and connect via eSATA or something? Are they fast enough?

"fast enough" depends on your needs. you've said you're on 100Mbit, though, so
i'd expect most decent eSATA enclosures to be fast enough to saturate that link.

i have no recommendations on specific enclosures.

anthony

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 210 bytes --]

Re: [9fans] Help with two small shared file servers

[John Floren]

On Wed, Aug 17, 2011 at 7:11 AM, Anthony Sorace <a@9srv.net> wrote:
> On Aug 17, 2011, at 6:09 AM, Aram Hăvărneanu wrote:
>
>> Can anyone shed some light on why I might want one and not the other?
>> Are there any other options?
>
> ken's fs is a kernel, and essentially gives you a 9p-accessible file storage
> appliance. it takes another box to be able to run it. #k is part of the standard
> cpu server kernel, and is typically used in conjunction with fossil and/or
> venti. really, you usually want to decide between Ken's fs and fossil (with or
> without venti). the basics:
>
> Ken's FS used to be the default file server in Plan 9. It's a kernel, and takes over
> a box. It is only directly accessible via 9p and il, which pretty much means via
> Plan 9 (cpu servers can share the resources as they like, of course). It does
> automatic daily archives of the whole system. It does limited de-duplicating.
> It is very fast and extremely stable.
>
> fossil is a newer file server, designed for use with venti (but usable on its own,
> as well). it can be configured to do daily archives and/or periodic ephemeral
> snapshots (you pick how often and how long they last). It can be used stand-
> alone (can't do archives that way, but can still do snapshots) or with venti, in
> which case you get venti's block-level deduplication (and direct access to
> block-level storage). fossil performs reasonably and is relatively stable, but
> less so than ken's fs (and, personally, seems more sensitive to unexpected
> shutdown). venti is very stable. typically, if fossil goes belly-up, you can
> reconstruct it from venti (in every case i've seen).
>
> personally, if you (a) have the extra box to spare, (b) can put in a little extra
> work up front, (c) don't care about direct access to block-level storage, and
> (d) can live without ephemeral snapshots, i'd suggest ken's fs; otherwise, i'd
> suggest fossil backed by venti.
>
>(note that (d) above is slightly tricky, as the ephemeral snapshots in fossil
>seem to trigger a bug that causes fossil to hang periodically; most reports
>have that happening every ~2 months or so)

Ken's FS serves only 9P and can be a hassle to set up and get going.
In my opinion, if you can live without ephemeral snapshots, just run
fossil+venti with snapshots turned off, which should eliminate the
stability complaint.

I ran Ken's FS for a while, but in the end it wasn't worth the extra
trouble. We're back with fossil + venti now, with fossil on the local
disk and Venti on a 16 TB Coraid.


>> Is 9p suitable for this? How will the 40ms latency affect 9p operation?
>
> i expect that'll be well low enough for most uses.
>

It's going to be slow. See my thesis
(https://bitbucket.org/floren/tstream/src/67c7419ad84a/documents/Thesis.pdf)
for details, but I'd expect file transfers to be at least 6 times
slower than transferring via HTTP. When I tested 9P vs. HTTP over
connections with 25 ms latency (50ms RTT), I saw a 4x slowdown versus
HTTP. Even at 15 ms RTT, transfers took twice as long.


John

Re: [9fans] Help with two small shared file servers

[erik quanstrom]

> > What's the best option for RAID in Plan9? I understand I can use
> > either Ken's fileserver or the Plan9 '#k' device.
>
> note that neither of these are RAID in the way most people expect. failure
> notification, in particular, can be lacking, and they're more restricted in what
> they try to do that many RAID systems. that said, I've used #k quite happily for
> years with no issues (and periodic correctness checks).

the way we use ken's file server is to connect it to
aoe storage which internally provides raid.  in fact,
we use the properties of the mirrors in the file server
to provide off site backup.  since the mirror always reads
from the first element which is very close, and since writes
are always done in the background and have no user
impact, the mirror doesn't change user-visable performance
at all.

- erik

Re: [9fans] Help with two small shared file servers

[Bakul Shah]

On Wed, 17 Aug 2011 13:09:47 +0300 =?UTF-8?B?QXJhbSBIxIN2xINybmVhbnU=?= <aram.h@mgk.ro>  wrote:
> Hello,
>
> I'm looking for advice on how to build a small network of two file
> servers. I'm hoping most servers to be Plan9, clients are Windows and
> Mac OS X.
>
> I have 2 houses separated by about 40ms of network latency. I want to
> set some servers in each location and have all data accessible from
> anywhere. I'll have about 2TB of data at each location, one location
> will probably scale up.
	...
> Is 9p suitable for this? How will the 40ms latency affect 9p
> operation? (I have 100Mbit).

With a strict request/response protocol you will get no more
than 64KB once every 80ms so your throughput at best will be
6.55Mbps or about 15 times slower than using HTTP/FTP on
100Mbps link for large files.  [John, what was the link speed
for the tests in your thesis?]

> Right now (only one location) I am using a Solaris server with ZFS
> that serves SMB and iSCSI.

Using venti in place of (or over) ZFS on spinning disks would
incur further performance degradation.

> Any tips are welcomed :-),

Since you want everything accessble from both sites, how about
temporarily caching remote files locally?  There was a usenix
paper about `nache', a caching proxy for nfs4 that may be of
interest. Or may be ftpfs with a local cache if remote access
is readonly?

Re: [9fans] Help with two small shared file servers

[John Floren]

On Wed, Aug 17, 2011 at 2:00 PM, Bakul Shah <bakul@bitblocks.com> wrote:
> On Wed, 17 Aug 2011 13:09:47 +0300 =?UTF-8?B?QXJhbSBIxIN2xINybmVhbnU=?= <aram.h@mgk.ro>  wrote:
>> Hello,
>>
>> I'm looking for advice on how to build a small network of two file
>> servers. I'm hoping most servers to be Plan9, clients are Windows and
>> Mac OS X.
>>
>> I have 2 houses separated by about 40ms of network latency. I want to
>> set some servers in each location and have all data accessible from
>> anywhere. I'll have about 2TB of data at each location, one location
>> will probably scale up.
>        ...
>> Is 9p suitable for this? How will the 40ms latency affect 9p
>> operation? (I have 100Mbit).
>
> With a strict request/response protocol you will get no more
> than 64KB once every 80ms so your throughput at best will be
> 6.55Mbps or about 15 times slower than using HTTP/FTP on
> 100Mbps link for large files.  [John, what was the link speed
> for the tests in your thesis?]
>

10 Mbps was the bottleneck--I had a 100 Mbit switch connected to one
side of the Linux gateway and a 10 Mbit hub on the other. I know a 10
Mbit hub is not modern networking equipment, but the only traffic on
it was the tests, and I also figured that 1. You're not likely to do
much better on a cross-country net link anyway, and 2. The HTTP tests
would suffer just as badly :)

If you'd like to Dare to Be Stupid, you can download my source and run
both servers with streaming enabled, then make some sort of
client-side caching filesystem which can stream from the remote (40 ms
away) server on behalf of applications, avoiding the necessity of
re-coding all your applications to understand streaming.

Or just serve your files over SMB, NFS, or HTTP.

I guess I didn't check, but if you *have* 2 TB of data but don't plan
to routinely *access* that much of it at any given time, you can
probably get away with using 9P.

I do not think there is any way a fossil+venti or kenfs file server
serving 9P will outperform your Solaris server. Not with our current
software. The way I use it, Plan 9 and 9P are "fast enough", but for
serious data transfer over such a high-latency link they won't cut it.
Plan 9 + HTTP would be a reasonable option if you just want to share
movies and music between your houses, I suppose.


John

Re: [9fans] Help with two small shared file servers

[Skip Tavakkolian]

if the link is stable, cfs(4) might be useful.

-Skip

On Wed, Aug 17, 2011 at 2:00 PM, Bakul Shah <bakul@bitblocks.com> wrote:
> On Wed, 17 Aug 2011 13:09:47 +0300 =?UTF-8?B?QXJhbSBIxIN2xINybmVhbnU=?= <aram.h@mgk.ro>  wrote:
>> Hello,
>>
>> I'm looking for advice on how to build a small network of two file
>> servers. I'm hoping most servers to be Plan9, clients are Windows and
>> Mac OS X.
>>
>> I have 2 houses separated by about 40ms of network latency. I want to
>> set some servers in each location and have all data accessible from
>> anywhere. I'll have about 2TB of data at each location, one location
>> will probably scale up.
>        ...
>> Is 9p suitable for this? How will the 40ms latency affect 9p
>> operation? (I have 100Mbit).
>
> With a strict request/response protocol you will get no more
> than 64KB once every 80ms so your throughput at best will be
> 6.55Mbps or about 15 times slower than using HTTP/FTP on
> 100Mbps link for large files.  [John, what was the link speed
> for the tests in your thesis?]
>
>> Right now (only one location) I am using a Solaris server with ZFS
>> that serves SMB and iSCSI.
>
> Using venti in place of (or over) ZFS on spinning disks would
> incur further performance degradation.
>
>> Any tips are welcomed :-),
>
> Since you want everything accessble from both sites, how about
> temporarily caching remote files locally?  There was a usenix
> paper about `nache', a caching proxy for nfs4 that may be of
> interest. Or may be ftpfs with a local cache if remote access
> is readonly?
>
>

Re: [9fans] Help with two small shared file servers

[erik quanstrom]

> > Is 9p suitable for this? How will the 40ms latency affect 9p
> > operation? (I have 100Mbit).
>
> With a strict request/response protocol you will get no more
> than 64KB once every 80ms so your throughput at best will be
> 6.55Mbps or about 15 times slower than using HTTP/FTP on
> 100Mbps link for large files.  [John, what was the link speed
> for the tests in your thesis?]

i calculate 1/0.080s/rt * 64*1024 = 819 kbytes/s.  (i think your
conversion from mbits/s to bytes/s is faulty.)

also you do get 1 oustanding per read or write, not per machine.
so the real limit is wire speed, but for most programs most of the
time the limit for a single file transfer would be as you specify.

- erik

Re: [9fans] Help with two small shared file servers

[erik quanstrom]

> Ken's FS serves only 9P and can be a hassle to set up and get going.
> In my opinion, if you can live without ephemeral snapshots, just run
> fossil+venti with snapshots turned off, which should eliminate the
> stability complaint.

since it's job is to be a plan 9 file server, this should be neither
surprising nor limiting.

> slower than transferring via HTTP. When I tested 9P vs. HTTP over
> connections with 25 ms latency (50ms RTT), I saw a 4x slowdown versus
> HTTP. Even at 15 ms RTT, transfers took twice as long.

did you do testing at regular lan latencies?  what i see on my networks
is usually <= 30µs.

- erik

Re: [9fans] Help with two small shared file servers

[Tristan Plumb]

> > > Is 9p suitable for this? How will the 40ms latency affect 9p
> > > operation? (I have 100Mbit).
> >
> > With a strict request/response protocol you will get no more
> > than 64KB once every 80ms so your throughput at best will be
> > 6.55Mbps or about 15 times slower than using HTTP/FTP on
> > 100Mbps link for large files.  [John, what was the link speed
> > for the tests in your thesis?]

> i calculate 1/0.080s/rt * 64*1024 = 819 kbytes/s.  (i think your
> conversion from mbits/s to bytes/s is faulty.)

819 kbyte/s just about than 6.55 Mbps, at 8 bits a byte.

and

64*1024/0.08 * 8 = 6553600

or are you talking about something else entirely?

tristan

--
All original matter is hereby placed immediately under the public domain.

Re: [9fans] Help with two small shared file servers

[John Floren]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

On Aug 17, 2011 10:40 PM, "erik quanstrom" <quanstro@quanstro.net> wrote:
> > slower than transferring via HTTP. When I tested 9P vs. HTTP over
> > connections with 25 ms latency (50ms RTT), I saw a 4x slowdown versus
> > HTTP. Even at 15 ms RTT, transfers took twice as long.
>
> did you do testing at regular lan latencies?  what i see on my networks
> is usually <= 30µs.
>
> - erik
>

I induced latencies similar to those found on the Internet. My tests at LAN
speeds (500us in my sub-optimal setup) had 9P at essentially the same
transfer rate as HTTP. My work was motivated by the observation that
transferring files to and from sources is just deathly slow, and that while
fcp does a good job helping that, it is just a patch covering a deeper
issue.

[-- Attachment #2: Type: text/html, Size: 924 bytes --]

Re: [9fans] Help with two small shared file servers

[Bakul Shah]

On Thu, 18 Aug 2011 01:29:29 EDT erik quanstrom <quanstro@labs.coraid.com>  wrote:
> > > Is 9p suitable for this? How will the 40ms latency affect 9p
> > > operation? (I have 100Mbit).
> >
> > With a strict request/response protocol you will get no more
> > than 64KB once every 80ms so your throughput at best will be
> > 6.55Mbps or about 15 times slower than using HTTP/FTP on
> > 100Mbps link for large files.  [John, what was the link speed
> > for the tests in your thesis?]
>
> i calculate 1/0.080s/rt * 64*1024 = 819 kbytes/s.  (i think your
> conversion from mbits/s to bytes/s is faulty.)

*8 for bits. 2^16*8/0.080/10^6 Mbps == 6.5536 Mbps.

MBps == 10^6 bytes/sec.
Mbps == 10^6 bits/sec.

But it is not accurate in another way. 40ms for the server to
receive the request. 40 ms for the client to receive the first
byte of data + time for transmission of 64KB == 2^16*8/10^8 ==
about 5ms. Actually a bit more due to ethernet, ip & tcp
overhead. So the total time will be over 85ms for a  BW of
under 6.17Mbps.

> also you do get 1 oustanding per read or write, not per machine.
> so the real limit is wire speed, but for most programs most of the
> time the limit for a single file transfer would be as you specify.

It is three things:

RTT= round trip time = 2*latency
BW = bandwidth
DS = datasize that can be sent without waiting.

Time to deliver DS bytes will be at least
	RTT+DS/BW

Consider three cases:
1) local (small RTT)
2) remote slow (large RTT, small BW)
3) remote fast (large RTT, large BW)

When RTT effects are small (1 & 2) 9p does fine. For case 3,
to minimize RTT effects you must make DS large but that can't
be done in 9p beyond 64k.

In John's test it was 10Mbps @ 25ms latency. To transfer 64KB
is 50ms rtt + 52ms or about 102ms. He should see 5.1Mbps or
about half the throughput compared to HTTP streaming case but
he says it was 1/4th so there must be other inefficiencies.

Re: [9fans] Help with two small shared file servers

[erik quanstrom]

> > did you do testing at regular lan latencies?  what i see on my networks
> > is usually <= 30µs.
> >
> > - erik
> >
>
> I induced latencies similar to those found on the Internet. My tests at LAN
> speeds (500us in my sub-optimal setup) had 9P at essentially the same
> transfer rate as HTTP. My work was motivated by the observation that
> transferring files to and from sources is just deathly slow, and that while
> fcp does a good job helping that, it is just a patch covering a deeper
> issue.

i think fcp gets right at the heart of the matter.  issuing multiple outstanding
messages is how protocols like aoe, pcie, etc. get speed.  writes are posted.

the question for me is how does one make this easy and natural.  one way
is to allow the mnt driver to issue multiple concurrent reads or writes for
the same i/o.  another would be to use a system call ring to allow programs
to have an arbitrary number of outstanding system calls.

- erik

Re: [9fans] Help with two small shared file servers

[Lucio De Re]

> issuing multiple outstanding
> messages is how protocols like aoe, pcie, etc. get speed.  writes are posted.
>
> the question for me is how does one make this easy and natural.  one way
> is to allow the mnt driver to issue multiple concurrent reads or writes for
> the same i/o.  another would be to use a system call ring to allow programs
> to have an arbitrary number of outstanding system calls.

The latter is the "windowing" feature that I seem to recall was first
introduced in HDLC or thereabouts (I'm no academic, I discovered this
feature many years after its invention).  But in those days
multithreading wasn't an option, maybe now it is possible to
consolidate both approaches into a single concept?  Both resemble
Inferno (Acme?) channels to me.

++L

Re: [9fans] Help with two small shared file servers

[cinap_lenrek]

%-W-NORML Normal Successful Completion

--
cinap :)

Re: [9fans] Help with two small shared file servers

[Ethan Grammatikidis]

On Wed, 17 Aug 2011 13:09:47 +0300
Aram Hăvărneanu <aram.h@mgk.ro> wrote:

> I have 2 houses separated by about 40ms of network latency. I want to
> set some servers in each location and have all data accessible from
> anywhere. I'll have about 2TB of data at each location, one location
> will probably scale up.

You might want to look into Octopus. It has another remote resource
protocol which doesn't support all the features of 9p but which
performs better in high-latency conditions. I couldn't give you the
details, it's been a while since I read about it.

http://lsub.org/ls/octopus.html

Re: [9fans] Help with two small shared file servers

[erik quanstrom]

> Right now (only one location) I am using a Solaris server with ZFS
> that serves SMB and iSCSI. It works great, there's nothing wrong with
> it, but I would like a Plan9 solution. Or do you think iSCSI would
> perform better with this latency? (I can't use AoE as AoE is not
> routable).

use a layer-2 vpn or gre.

- erik