Re: [9fans] Fossil fs recovery

Re: [9fans] Fossil fs recovery

[Bernd Maier]

Done with real hardware today. Works fine.

-Bernd


On Fri, 12 Aug 2011 11:53:09 +0200 Bernd Maier <bema@hush.com>
wrote:
>Many thanks! I will try it this weekend.
>
>One additional question:
>
>How much RAM needs the Fossil+Venti Fileserver in total?
>
>-Bernd
>
>On Thu, 11 Aug 2011 10:11:27 +0200 David du Colombier
><0intro@gmail.com> wrote:
>>In fact, it's pretty straightforward.
>>
>>In the following example, I consider you have a disk sdE0
>>with Fossil, and you want to add a disk sdE1 with Venti.
>>
>>Create the Venti partitions:
>>
>>disk/fdisk -baw /dev/sdE1/data
>>disk/prep -bw -a^(isect arenas bloom) /dev/sdE1/plan9
>>
>>Format the Venti file systems:
>>
>>venti/fmtisect isect /dev/sdE1/isect
>>venti/fmtarenas arenas /dev/sdE1/arenas
>>venti/fmtbloom /dev/sdE1/bloom
>>
>>Since the bloom filter requires memory equal to its size
>>on disk, you could need to pass the parameter -s to fmtbloom
>>to specify its size or directly adjust the partition size.
>>Keep in mind that Venti use 20% total memory by default, and
>>1/3 of it is allocated to the bloom filter.
>>For example, with 1 GB of memory, you cannot have a bloom filter
>>larger than 64 MB (which is sufficient) with the default
>settings.
>>Of course, the bloom filter is optional anyway.
>>
>>Configure Venti:
>>
>>echo 'index main
>>isect /dev/sdE1/isect
>>arenas /dev/sdE1/arenas
>>bloom /dev/sdE1/bloom' | venti/conf -w /dev/sdE1/arenas
>>
>>Initialize the index sections:
>>
>>venti/fmtindex /dev/sdE1/arenas
>>
>>Read your current Fossil configuration:
>>
>>fossil/conf /dev/sdE0/fossil > fossil.conf
>>
>>Edit fossil.conf, it should look like something like:
>>
>>fsys main config /dev/sdE0/fossil
>>fsys main open -V -c 3000
>>
>>Remove -V to open and srv if needed, then
>>add the following line to enable automatic
>>snapshots:
>>
>>fsys main snaptime -s 60 -a 0530 -t 2880
>>
>>(snapshot at 05:30, temporary snapshot each
>> hour and discarded after two days)
>>
>>Write your new Fossil configuration:
>>
>>fossil/conf -w /dev/sdE0/fossil < fossil.conf
>>
>>Mount your 9fat file system, then add the following line
>>to the file /n/9fat/plan9.ini:
>>
>>venti=#S/sdE1/arenas
>>
>>After rebooting, the current Fossil content will be dumped
>>to Venti.
>>
>>--
>>David du Colombier

Re: [9fans] Fossil fs recovery

[David du Colombier]

> I certinaly had problems a few years ago with fossil deadlocking,
> since I disabled ephemeral snapshots it has been quite reliable.
>
> My feeling (nothing more than that) was that the deadlocks seemed to
> occur when I receved email just at the time of the snapshot.
>
> the problem seemed to occur at a rate of once per month or two
> for my lightly loaded home server.
>
> I did try to run two file servers for a while, exporting
> /proc from the test machine with snapshots enabled to a seccond one.
> The hope was that I would be able to use acid to debug the test
> machine's fossil when the problem occured. Sadly I never caught the
> problem and had to decomission the old machine (house repairs).

It is pretty easy to reproduce this bug.

When writing a large file, run "snap -a", then "snap" four times.
Fossil should hang.

I just tried and I am able to reproduce it reliably on a new Fossil.

I've uploaded the stack traces on my website.

http://www.9grid.fr/www.9grid.fr/misc/fossil/deadlock1

--
David du Colombier

Re: [9fans] Fossil fs recovery

[Francisco J Ballesteros]

At lsub we had deadlocks when temporary snapshots were
enable, as others. Also, fsys all sync (even with temporary snaps disabled)
sometimes hang the fossil and it no longer respond. I couldn't track this down,
we just sync fsys one by one.

hth

Re: [9fans] Fossil fs recovery

[Steve Simon]

I certinaly had problems a few years ago with fossil deadlocking,
since I disabled ephemeral snapshots it has been quite reliable.

My feeling (nothing more than that) was that the deadlocks seemed to
occur when I receved email just at the time of the snapshot.

the problem seemed to occur at a rate of once per month or two
for my lightly loaded home server.

I did try to run two file servers for a while, exporting
/proc from the test machine with snapshots enabled to a seccond one.
The hope was that I would be able to use acid to debug the test machine's
fossil when the problem occured. Sadly I never caught the problem and
had to decomission the old machine (house repairs).

-Steve

Re: [9fans] Fossil fs recovery

[Lyndon Nerenberg (VE6BBM/VE7TFX)]

> Actually we have since found that even scheduling archival
> and temporary snapshots at different times isn't always
> enough to prevent fossil from hanging during archival
> snapshots.

What size of fossil and venti?  Can you post details of how
the servers are set up?  I.e. fossil/conf and venti/conf plus
the output from 'hget http://venti/storage'?  Maybe we can
find a pattern in the configuration of these setups that's
provoking the hangs.

Any idea how long it's taking your archive dumps to run out?
And how much data is being flushed?

For that matter, can anyone explain the use of those
undecipherable venti graph/... urls?  Maybe at least
the victims can start logging some performance numbers
for more pattern matching?

--lyndon

Re: [9fans] Fossil fs recovery

[geoff]

Actually we have since found that even scheduling archival
and temporary snapshots at different times isn't always
enough to prevent fossil from hanging during archival
snapshots.

Re: [9fans] Fossil fs recovery

[David du Colombier]

> How much RAM needs the Fossil+Venti Fileserver in total?

For both Fossil and Venti, the default, in Plan 9, is to
use 20% of total user memory. These values are defined in
/sys/src/9/boot/local.c.

The minimum required memory is something like 1 MB for Fossil
and 45 MB for Venti.

Venti have three caches (lump, block and index) and Fossil have a
single cache. Their size can be specified either manually in their
own configuration or automatically with the -m parameter.
Beware that for Venti, 1/3 of this percentage should be able to
contain the Bloom filter.

--
David du Colombier

Re: [9fans] Fossil fs recovery

[Bernd Maier]

Many thanks! I will try it this weekend.

One additional question:

How much RAM needs the Fossil+Venti Fileserver in total?

-Bernd

On Thu, 11 Aug 2011 10:11:27 +0200 David du Colombier
<0intro@gmail.com> wrote:
>In fact, it's pretty straightforward.
>
>In the following example, I consider you have a disk sdE0
>with Fossil, and you want to add a disk sdE1 with Venti.
>
>Create the Venti partitions:
>
>disk/fdisk -baw /dev/sdE1/data
>disk/prep -bw -a^(isect arenas bloom) /dev/sdE1/plan9
>
>Format the Venti file systems:
>
>venti/fmtisect isect /dev/sdE1/isect
>venti/fmtarenas arenas /dev/sdE1/arenas
>venti/fmtbloom /dev/sdE1/bloom
>
>Since the bloom filter requires memory equal to its size
>on disk, you could need to pass the parameter -s to fmtbloom
>to specify its size or directly adjust the partition size.
>Keep in mind that Venti use 20% total memory by default, and
>1/3 of it is allocated to the bloom filter.
>For example, with 1 GB of memory, you cannot have a bloom filter
>larger than 64 MB (which is sufficient) with the default settings.
>Of course, the bloom filter is optional anyway.
>
>Configure Venti:
>
>echo 'index main
>isect /dev/sdE1/isect
>arenas /dev/sdE1/arenas
>bloom /dev/sdE1/bloom' | venti/conf -w /dev/sdE1/arenas
>
>Initialize the index sections:
>
>venti/fmtindex /dev/sdE1/arenas
>
>Read your current Fossil configuration:
>
>fossil/conf /dev/sdE0/fossil > fossil.conf
>
>Edit fossil.conf, it should look like something like:
>
>fsys main config /dev/sdE0/fossil
>fsys main open -V -c 3000
>
>Remove -V to open and srv if needed, then
>add the following line to enable automatic
>snapshots:
>
>fsys main snaptime -s 60 -a 0530 -t 2880
>
>(snapshot at 05:30, temporary snapshot each
> hour and discarded after two days)
>
>Write your new Fossil configuration:
>
>fossil/conf -w /dev/sdE0/fossil < fossil.conf
>
>Mount your 9fat file system, then add the following line
>to the file /n/9fat/plan9.ini:
>
>venti=#S/sdE1/arenas
>
>After rebooting, the current Fossil content will be dumped
>to Venti.
>
>--
>David du Colombier

Re: [9fans] Fossil fs recovery

[David du Colombier]

> > Is it obvious enough from the man pages that this wouldn't be too
> > useful to have on the Plan 9 wiki?
> >
> > I'm a big believer in the wiki, but not when it pushes one to avoid
> > reading the authoritative documentation of the man pages.
>
> http://www.plan9.bell-labs.com/wiki/plan9/Setting_up_Venti/index.html
>
> has the gist of setting up venti by hand.  The bits on re-configuring
> Fossil aren't there (save for the mention of the venti environment
> variable).  In my mind the man pages pick up the slack of the wiki in
> this case.

This wiki page seems a little outdated, probably because it was written
before the new Venti came out, and probably even before Fossil.
For example, there is no mention of the Bloom filter.

Feel free to complete it if you wish. Don't hesitate to paraphrase my
precedent message if you want.

--
David du Colombier

Re: [9fans] Fossil fs recovery

[David du Colombier]

> > fsys main snaptime -s 60 -a 0530 -t 2880
>
> Beware: there is currently a bug in fossil which can cause
> it to deadlock under load if ephemerial snapshots are enabled.
>
> This I reccomend you use a configuration more like:
>
> 	fsys main snaptime -s none -a 0530 -t none

As far as know and like Geoff Collyer wrote to the fossilcons(8)
manual recently, this bug happens when both archival and
temporary snapshots happen at the same time.

This is not the case here, because the archival snapshot is
taken at 05:30, while the temporary snapshot happens on the hour.

Thus, it should be safe to use temporary snapshots here.

--
David du Colombier

Re: [9fans] Fossil fs recovery

[Steve Simon]

> fsys main snaptime -s 60 -a 0530 -t 2880

Beware: there is currently a bug in fossil which can cause
it to deadlock under load if ephemerial snapshots are enabled.

This I reccomend you use a configuration more like:

	fsys main snaptime -s none -a 0530 -t none

-Steve

Re: [9fans] Fossil fs recovery

[Jack Norton]

David Leimbach wrote:
> Is it obvious enough from the man pages that this wouldn't be too useful
> to have on the Plan 9 wiki?
>
> I'm a big believer in the wiki, but not when it pushes one to avoid
> reading the authoritative documentation of the man pages.
>
> Dave
>
http://www.plan9.bell-labs.com/wiki/plan9/Setting_up_Venti/index.html

has the gist of setting up venti by hand.  The bits on re-configuring
Fossil aren't there (save for the mention of the venti environment
variable).  In my mind the man pages pick up the slack of the wiki in
this case.

Also, I personally use the 9fans archive as my own personal manual next
to the man pages (and at last resort the wiki).  I for some reason
always get the impression that the wiki is missing bits... so I use it
only has a means of discovering just what man page exactly I should be
reading :)

-Jack

Re: [9fans] Fossil fs recovery

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

Is it obvious enough from the man pages that this wouldn't be too useful to
have on the Plan 9 wiki?

I'm a big believer in the wiki, but not when it pushes one to avoid reading
the authoritative documentation of the man pages.

Dave

On Thu, Aug 11, 2011 at 1:11 AM, David du Colombier <0intro@gmail.com>wrote:

> In fact, it's pretty straightforward.
>
> In the following example, I consider you have a disk sdE0
> with Fossil, and you want to add a disk sdE1 with Venti.
>
> Create the Venti partitions:
>
> disk/fdisk -baw /dev/sdE1/data
> disk/prep -bw -a^(isect arenas bloom) /dev/sdE1/plan9
>
> Format the Venti file systems:
>
> venti/fmtisect isect /dev/sdE1/isect
> venti/fmtarenas arenas /dev/sdE1/arenas
> venti/fmtbloom /dev/sdE1/bloom
>
> Since the bloom filter requires memory equal to its size
> on disk, you could need to pass the parameter -s to fmtbloom
> to specify its size or directly adjust the partition size.
> Keep in mind that Venti use 20% total memory by default, and
> 1/3 of it is allocated to the bloom filter.
> For example, with 1 GB of memory, you cannot have a bloom filter
> larger than 64 MB (which is sufficient) with the default settings.
> Of course, the bloom filter is optional anyway.
>
> Configure Venti:
>
> echo 'index main
> isect /dev/sdE1/isect
> arenas /dev/sdE1/arenas
> bloom /dev/sdE1/bloom' | venti/conf -w /dev/sdE1/arenas
>
> Initialize the index sections:
>
> venti/fmtindex /dev/sdE1/arenas
>
> Read your current Fossil configuration:
>
> fossil/conf /dev/sdE0/fossil > fossil.conf
>
> Edit fossil.conf, it should look like something like:
>
> fsys main config /dev/sdE0/fossil
> fsys main open -V -c 3000
>
> Remove -V to open and srv if needed, then
> add the following line to enable automatic
> snapshots:
>
> fsys main snaptime -s 60 -a 0530 -t 2880
>
> (snapshot at 05:30, temporary snapshot each
>  hour and discarded after two days)
>
> Write your new Fossil configuration:
>
> fossil/conf -w /dev/sdE0/fossil < fossil.conf
>
> Mount your 9fat file system, then add the following line
> to the file /n/9fat/plan9.ini:
>
> venti=#S/sdE1/arenas
>
> After rebooting, the current Fossil content will be dumped
> to Venti.
>
> --
> David du Colombier
>
>

[-- Attachment #2: Type: text/html, Size: 2810 bytes --]

Re: [9fans] Fossil fs recovery

[David du Colombier]

In fact, it's pretty straightforward.

In the following example, I consider you have a disk sdE0
with Fossil, and you want to add a disk sdE1 with Venti.

Create the Venti partitions:

disk/fdisk -baw /dev/sdE1/data
disk/prep -bw -a^(isect arenas bloom) /dev/sdE1/plan9

Format the Venti file systems:

venti/fmtisect isect /dev/sdE1/isect
venti/fmtarenas arenas /dev/sdE1/arenas
venti/fmtbloom /dev/sdE1/bloom

Since the bloom filter requires memory equal to its size
on disk, you could need to pass the parameter -s to fmtbloom
to specify its size or directly adjust the partition size.
Keep in mind that Venti use 20% total memory by default, and
1/3 of it is allocated to the bloom filter.
For example, with 1 GB of memory, you cannot have a bloom filter
larger than 64 MB (which is sufficient) with the default settings.
Of course, the bloom filter is optional anyway.

Configure Venti:

echo 'index main
isect /dev/sdE1/isect
arenas /dev/sdE1/arenas
bloom /dev/sdE1/bloom' | venti/conf -w /dev/sdE1/arenas

Initialize the index sections:

venti/fmtindex /dev/sdE1/arenas

Read your current Fossil configuration:

fossil/conf /dev/sdE0/fossil > fossil.conf

Edit fossil.conf, it should look like something like:

fsys main config /dev/sdE0/fossil
fsys main open -V -c 3000

Remove -V to open and srv if needed, then
add the following line to enable automatic
snapshots:

fsys main snaptime -s 60 -a 0530 -t 2880

(snapshot at 05:30, temporary snapshot each
 hour and discarded after two days)

Write your new Fossil configuration:

fossil/conf -w /dev/sdE0/fossil < fossil.conf

Mount your 9fat file system, then add the following line
to the file /n/9fat/plan9.ini:

venti=#S/sdE1/arenas

After rebooting, the current Fossil content will be dumped
to Venti.

--
David du Colombier

Re: [9fans] Fossil fs recovery

[ron minnich]

On Wed, Aug 10, 2011 at 2:58 PM, Steve Simon <steve@quintile.net> wrote:
> You can migrate from fossil to fossil and venti
> but it's a faf and takes quite a bit of care/reading manuals;
> this is how I did it the first time I installed fossil.

um.

When I've done it it's been as simple as having a venti there and a
venti  environment variable ready. So if you have stuff you do not
want to lose do not despair.

Unless I got confused.

ron

Re: [9fans] Fossil fs recovery

[Steve Simon]

You can migrate from fossil to fossil and venti
but it's a faf and takes quite a bit of care/reading manuals;
this is how I did it the first time I installed fossil.

reinstalling from scratch is probably easier if your fossil is
only an experimental system.

-Steve

Re: [9fans] Fossil fs recovery

[David du Colombier]

> I am not sure if this works without loosing all data on the Fossil
> filesystem.

You will not lose anything.

You need to create the Venti partitions, format them, configure Venti,
remove -V in Fossil configuration and adapt your plan9.ini.

After rebooting, your Fossil will start to dump to Venti.

--
David du Colombier

Re: [9fans] Fossil fs recovery

[Bernd Maier]

On Wed, 10 Aug 2011 17:25:53 +0200 Steve Simon <steve@quintile.net>
wrote:


>This is only valid if you are running venti and have have fossil
>and venti on seperate physical disks of course.

Good point. I did an installation with Fossil only. I will try to
add
a disk with Venti next. I am not sure if this works without loosing
all data on the Fossil filesystem. Acutally I am working in a
virtualbox enviroment for testing purpose. When I am ready to
switch
to real hardware I would like to use backup to blue ray media too.

Regards,
Bernd

Re: [9fans] Fossil fs recovery

[Steve Simon]

> con -l /srv/fossil
> prompt: fsys main
> main: check fix
> ...

that looks plausable though its from memory,
I have not checked the manual pages in detail.

the error you saw at the start:

	iostate is 2 in blockwrite

looks very reminescent of when I had a real
hardware failure on a physical disk.

I was lucky as I was running fossil and venti so I did this:

	reboot onto the live CDROM.
	use fossil/last to get the last good venti score
	shutdown
	swap fossil for a new disk
	reboot the live cdrom
	disk/fdisk, disk/prep, disk/mbr etc of the new disk
	start venti
	reinitialised the new fossil partition from the last good venti score
	reboot

This is only valid if you are running venti and have have fossil
and venti on seperate physical disks of course.

-Steve

[9fans] Fossil fs recovery

[Bernd Maier]

Hello,

today there were problems with the Fossil FS on my Plan9 test
installation. I got first something like:

"iostate is 2 in blockwrite"

I did than the following steps.

I would like to ask if this is the right way to fix a Fossil FS? I
am new to
Plan9 but I come from the Unix world and this was close to my Unix
experience.

Regards,
Bernd


What I did:

con -l /srv/fossil
prompt: fsys main
main: check

main: check
checking epoch 1...
check: visited 1/1377184 blocks (0%)
check: visited 13772/1377184 blocks (1%)
check: visited 27543/1377184 blocks (2%)
check: visited 41314/1377184 blocks (3%)
check: visited 55085/1377184 blocks (4%)
fsys blocks: total=1377184 used=63411(4.6%) free=1308340(95.0%)
lost=5433(0.4%)

error: source has been removed: /active/sys/src/fs/wrens/wrens.8 ->
72
error: source has been removed: /active/sys/src/fs/wrens/uidgid.8 -
> 47
error: source has been removed: /active/sys/src/fs/wrens/udp.8 ->
62
error: source has been removed: /active/sys/src/fs/wrens/trap.8 ->
25
error: source has been removed: /active/sys/src/fs/wrens/toy.8 ->
24
error: source has been removed: /active/sys/src/fs/wrens/timezone.h
-> 46
error: source has been removed: /active/sys/src/fs/wrens/time.8 ->
71
error: source has been removed: /active/sys/src/fs/wrens/sub.8 ->
45
error: source has been removed: /active/sys/src/fs/wrens/sntp.8 ->
61
error: source has been removed: /active/sys/src/fs/wrens/rawcopy.8 -
> 44
error: source has been removed: /active/sys/src/fs/wrens/proc.8 ->
43
error: source has been removed: /active/sys/src/fs/wrens/print.8 ->
42
error: source has been removed: /active/sys/src/fs/wrens/pci.8 ->
23
error: source has been removed: /active/sys/src/fs/wrens/pc.8 -> 22
error: source has been removed: /active/sys/src/fs/wrens/part.8 ->
54
...

main: check fix
...
error: source has been removed: /active/sys/src/fs/wrens/cga.8 ->
14
error: could not open source: /active/sys/src/fs/wrens/bootflp ->
75: corrupted file entry
error: source has been removed: /active/sys/src/fs/wrens/auth.8 ->
27
error: source has been removed: /active/sys/src/fs/wrens/ata.8 ->
13
error: source has been removed: /active/sys/src/fs/wrens/arp.8 ->
55
error: source has been removed: /active/sys/src/fs/wrens/aoe.8 ->
48
error: source has been removed:
/active/sys/src/fs/wrens/9wrensfs.gz -> 74
error: source has been removed: /active/sys/src/fs/wrens/9wrensfs -
> 73
error: source has been removed: /active/sys/src/fs/wrens/9p2.8 ->
26
error: source has been removed: /active/sys/src/fs/wrens/8253.8 ->
12
error: source has been removed: /active/sys/src/fs/wrens/8250.8 ->
11
fsck: 65 clri, 0 clre, 0 clrp, 5433 bclose


Next check:

main: check
checking epoch 1...
check: visited 1/1377184 blocks (0%)
check: visited 13772/1377184 blocks (1%)
check: visited 27543/1377184 blocks (2%)
check: visited 41314/1377184 blocks (3%)
check: visited 55085/1377184 blocks (4%)
fsys blocks: total=1377184 used=63411(4.6%) free=1313773(95.4%)
lost=0(0.0%)
error: non referenced entry in source /active/sys/src/fs/wrens[10]
error: non referenced entry in source /active/sys/src/fs/wrens[11]
error: non referenced entry in source /active/sys/src/fs/wrens[12]
error: non referenced entry in source /active/sys/src/fs/wrens[13]
error: non referenced entry in source /active/sys/src/fs/wrens[14]
error: non referenced entry in source /active/sys/src/fs/wrens[15]
error: non referenced entry in source /active/sys/src/fs/wrens[16]
error: non referenced entry in source /active/sys/src/fs/wrens[17]
error: non referenced entry in source /active/sys/src/fs/wrens[18]
error: non referenced entry in source /active/sys/src/fs/wrens[19]
error: non referenced entry in source /active/sys/src/fs/wrens[20]
error: non referenced entry in source /active/sys/src/fs/wrens[21]
error: non referenced entry in source /active/sys/src/fs/wrens[22]
error: non referenced entry in source /active/sys/src/fs/wrens[23]
error: non referenced entry in source /active/sys/src/fs/wrens[24]
error: non referenced entry in source /active/sys/src/fs/wrens[25]
error: non referenced entry in source /active/sys/src/fs/wrens[26]
error: non referenced entry in source /active/sys/src/fs/wrens[27]
error: non referenced entry in source /active/sys/src/fs/wrens[28]
error: non referenced entry in source /active/sys/src/fs/wrens[29]
error: non referenced entry in source /active/sys/src/fs/wrens[30]
error: non referenced entry in source /active/sys/src/fs/wrens[31]
...
error: non referenced entry in source /active/sys/src/fs/wrens[74]
fsck: 0 clri, 64 clre, 0 clrp, 0 bclose


Stop system

cpu% fshalt
syncing.../srv/fscons...
main: venti...
halting.../srv/fscons...fsys all sync
main sync: wrote 7 blocks
done halting

Restart again

cpu% con -l /srv/fscons
prompt: fsys main
main: check
checking epoch 1...
check: visited 1/1377184 blocks (0%)
check: visited 13772/1377184 blocks (1%)
check: visited 27543/1377184 blocks (2%)
check: visited 41314/1377184 blocks (3%)
check: visited 55085/1377184 blocks (4%)
fsys blocks: total=1377184 used=63217(4.6%) free=1313773(95.4%)
lost=194(0.0%)
fsck: 0 clri, 0 clre, 0 clrp, 194 bclose