Re: [9fans] Fossil+Venti on Linux

[erik quanstrom <quanstro@quanstro.net>]

> You could adapt Plan B's bns to fail over between different FSs. But...
> We learned that although you can let he FS fail over nicely, many other
> things stand in the way making it unnecessary to fail over. For example,
> on Plan 9, cs and dns have problems after a fail over, your IP address
> may change, etc. All that is to say that when you experience tolerance
> to FS failures, you still face other things that do not fail over.
>
> To tolerate failures what we do is to run venti on
> a raid. If fossil gets corrupted somehow we'd just format the partition
> using the last vac. To survive crashes of the machine with the venti we
> copy its arenas to another machine, aso keeping a raid.

forgive a bit of off-topicness.  this is about ken's filesystem, not
venti or fossil.

the coraid fs maintains its cache on a local AoE-based raid10 and it
automaticlly mirrors its worm on two AoE-based raid5 targets.  the
secondary worm target is in a seperate building with a backup fs.
since reads always start with the first target, the slow offsite link
is not noticed.  (we frequently exceed the bandwidth of the
backup link -- now 100Mbps --- to the cache, so replicating the cache
would be impractical.)

we can sustain the loss of a disk drive with only a small and
temporary performance hit.  the storage targets may be rebooted with a
small pause in service.

more severe machine failues can be recovered with varing degrees of
pain.  only if both raid targets were lost simultainously would more than
24hrs of data be lost.

we don't do any failover.  we try to keep the fs up instead.  we have
had two unplanned fs outages in 2 years.  one was due to a corrupt
sector leading to a bad tag.  the other was a network problem due to
an electrical storm that could have been avoided if i'd been on the
ball.

the "diskless fileserver" paper from iwp9 has the gory details.

- erik