[9fans] venti backup

[9fans] venti backup

[hugo rivera]

Hello,
I've backed up all my *active* arenas in to another disk, just to be
safe. The man page says that the index and the bloom filter may be
rebuilt if lost. So it seems sufficient to backup my arenas, am I
right?
saludos,

--
Hugo

Re: [9fans] venti backup

[Steve Simon]

> So it seems sufficient to backup my arenas, am I
> right?

Yes, exactly, I haev done this several times.

It might take a few hours and some studying of manuals
but the arenas are all you need.

-Steve

Re: [9fans] venti backup

[hugo rivera]

Great. Thanks.
I hope I will never need to use my backed up arenas :-)

2012/1/17 Steve Simon <steve@quintile.net>:
>> So it seems sufficient to backup my arenas, am I
>> right?
>
> Yes, exactly, I haev done this several times.
>
> It might take a few hours and some studying of manuals
> but the arenas are all you need.
>
> -Steve
>



--
Hugo

Re: [9fans] venti backup

[David du Colombier]

> I've backed up all my *active* arenas in to another disk, just to be
> safe. The man page says that the index and the bloom filter may be
> rebuilt if lost. So it seems sufficient to backup my arenas, am I
> right?

Yes, you can rebuild the index and the Bloom filter
with 'venti/buildindex -b'.

--
David du Colombier

Re: [9fans] venti backup

[hugo rivera]

Just to make sure I could rebuild things in case I should, I've tried
to recover everything from my backed up arenas, but I failed. I am not
sure if things go wrong because the backup per se is wrong or I am
making a mistake while recovering from the backup (or both).
So this is how I create the back up of my active arenas

% venti/checkarenas -v /dev/da1s4
output supresed
arena='arenas059' [31676186624,32213057536)
        version=5 created=1326893834 modified=1326893851
        clumps=34,617 compressed clumps=17,908 data=274,318,099
compressed data=198,689,879 disk storage=200,870,750
% 9 dd -if /dev/da1s4 -of fa -bs 8192 -count 3932258     #
8192*3932258 = 32213057536

Then I create a zeroed file for the bloom filter and another for the
index section. After I format those files I run

% venti/buildindex -b venti.conf
2012/0119 17:11:41 err 2: invalid ending address in arena table
venti/buildindex: can't init venti: can't initialize venti:
/home/hugo/tmpventi/fa: invalid ending address in arena table

It seems that the backup I create is not correct, am I right?

2012/1/17 David du Colombier <0intro@gmail.com>:
>> I've backed up all my *active* arenas in to another disk, just to be
>> safe. The man page says that the index and the bloom filter may be
>> rebuilt if lost. So it seems sufficient to backup my arenas, am I
>> right?
>
> Yes, you can rebuild the index and the Bloom filter
> with 'venti/buildindex -b'.
>
> --
> David du Colombier
>



--
Hugo

Re: [9fans] venti backup

[David du Colombier]

> It seems that the backup I create is not correct, am I right?

You truncated your arena partition. The new partition size
doesn't match the size specified in ArenaPart.

You should format a new arena partition, then copy arenas
from the beginning of the first to the end of the last.

--
David du Colombier

Re: [9fans] venti backup

[David du Colombier]

To clarify things.

You backup is correct, but it's not necessary to backup the
first 128 blocks of the arena partition. Its only contains
the Venti configuration and the ArenaPart structure.

Here is an example of what I described in my precedent message.

Create an arena partition at least as big as your original
one, and format it:

% dd -if /dev/zero -of arenas.img -bs 8192 -count 4194304
% venti/fmtarenas arenas arenas.img

Then copy your Venti arenas from the beginning of arenas00
(128*8192) to the end of arenas059 (3932258*8192):

% dd -if /dev/da1s4 -of arenas.img -bs 8192 -seek 128 -oseek 128 -count 3932258

Of course, '/dev/da1s4' can be 'fa' in your example.
If you removed the first 128 blocks, you don't have
to use '-seek'.

Finally, write the Venti configuration, rebuild the index
and Bloom filter, and start Venti.

--
David du Colombier

Re: [9fans] venti backup

[hugo rivera]

There's something weird going on. First checkarenas reports

% venti/checkarenas -v /dev/da1s4
arena='arenas00' [802816,537673728)
        version=5 created=1265030300 modified=1265248834 sealed
        score=f383ebf9edefe8d37733c8caba6ff53e8b5517b0
        clumps=82,908 compressed clumps=22,812 data=669,897,790
compressed data=531,617,136 disk storage=536,840,340

that's only 98 blocks of 8192 bytes, not 128 as you mention.
Anyway, I run fmtarenas on fa and then if I

% 9 dd -if /dev/da1s4 -of fa -bs 8192 -iseek 98 -oseek 98 -count 65536

or

% # this doesn't make sense but I've tried it, nevertheless
% 9 dd -if /dev/da1s4 -of fa -bs 8192 -iseek 128 -oseek 128 -count 65634

I get

2012/0120 11:57:10 err 2: arena set has wrong magic number: 00000000
expected ArenaPartMagic (0xa9e4a5e7)
venti/buildindex: can't init venti: can't initialize venti: fa: arena
set has wrong magic number: 00000000 expected ArenaPartMagic
(0xa9e4a5e7)

when I run buildindex, and checkarenas says the same thing.


2012/1/20 David du Colombier <0intro@gmail.com>:
> To clarify things.
>
> You backup is correct, but it's not necessary to backup the
> first 128 blocks of the arena partition. Its only contains
> the Venti configuration and the ArenaPart structure.
>
> Here is an example of what I described in my precedent message.
>
> Create an arena partition at least as big as your original
> one, and format it:
>
> % dd -if /dev/zero -of arenas.img -bs 8192 -count 4194304
> % venti/fmtarenas arenas arenas.img
>
> Then copy your Venti arenas from the beginning of arenas00
> (128*8192) to the end of arenas059 (3932258*8192):
>
> % dd -if /dev/da1s4 -of arenas.img -bs 8192 -seek 128 -oseek 128 -count 3932258
>
> Of course, '/dev/da1s4' can be 'fa' in your example.
> If you removed the first 128 blocks, you don't have
> to use '-seek'.
>
> Finally, write the Venti configuration, rebuild the index
> and Bloom filter, and start Venti.
>
> --
> David du Colombier
>



--
Hugo

Re: [9fans] venti backup

[David du Colombier]

> that's only 98 blocks of 8192 bytes, not 128 as you mention.

Sorry, I got confused. It's 98 blocks on arena partition and
128 blocks on isect partition.

I just tried. This is what I did.

The goal is to manually recopy the first arena from
the first Venti (arenas1.img) to the second Venti (arenas2.img).

It will work for you as long as you adjust the partition
names, the number of arenas and the sizes.

# first venti

% dd -if /dev/zero -of arenas1.img -bs 8192 -count 131072
% dd -if /dev/zero -of isect1.img -bs 8192 -count 16384
% dd -if /dev/zero -of bloom1.img -bs 8192 -count 8192
% venti/fmtarenas arenas arenas1.img
% venti/fmtisect isect isect1.img
% venti/fmtbloom bloom1.img
% echo 'index main
    isect isect1.img
    arenas arenas1.img
    bloom bloom1.img' | venti/conf -w arenas1.img
% venti/fmtindex arenas1.img
% venti=127.1
% venti/venti -c arenas1.img -m 20 -h tcp!127.1!8000
% vac /sys/src/9/pc
% hget http://127.1:8000/index
index=main version=1 blocksize=8192 tabsize=524288
    buckets=16287 div=263706
    sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
    arena='arenas0' on arenas1.img at [802816,537673728)
    version=5 created=1327069595 modified=1327069627
    written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
    indexed: clumps=0 compressed clumps=0 data=0 compressed data=0 storage=0
% venti/sync
% Kill venti | rc
% bc
802816/8192 = 98
537673728/8192 = 65634
65634-98 = 65536

# second venti

% dd -if /dev/zero -of arenas2.img -bs 8192 -count 65633
% dd -if /dev/zero -of isect2.img -bs 8192 -count 16384
% dd -if /dev/zero -of bloom2.img -bs 8192 -count 8192
% venti/fmtarenas arenas arenas2.img
% venti/fmtisect isect isect2.img
% venti/fmtbloom bloom2.img
% echo 'index main
    isect isect2.img
    arenas arenas2.img
    bloom bloom2.img' | venti/conf -w arenas2.img
% venti/fmtindex arenas2.img
% dd -if arenas2.img -of arenapart -bs 8192 -count 97
% dd -if arenas1.img -of arenas -bs 8192 -iseek 97 -count 65536
% cat arenapart arenas > arenas2.img
% rm -f arenapart arenas
% venti/buildindex -b arenas2.img
0 clumps, 16,287 buckets
2012/0120 14:29:29 read index
venti/buildindex: brand-new index, no work to do
2012/0120 14:29:29 arena arenas0: 669 entries
% venti=127.1
% venti/venti -c arenas2.img -m 20 -h tcp!127.1!8000
term% hget http://127.1:8000/index
index=main version=1 blocksize=8192 tabsize=524288
    buckets=16287 div=263706
    sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
    arena='arenas0' on arenas2.img at [802816,537673728)
    version=5 created=1327069595 modified=1327069627
    written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
    indexed: clumps=0 compressed clumps=0 data=0 compressed data=0
    storage=0

Honestly, I think you should just use venti/wrarena to write
backed up arenas to a running Venti. It's much easier.

--
David du Colombier

Re: [9fans] venti backup

[hugo rivera]

Thanks a lot, David, for your detailed reply.
I've followed your indications and now I am able to recover from my
venti backup :-)
I must confess that I am puzzled, because some sizes and most seeks
for dd are off by 1 block from what I expect. Particulary,
why do you
% dd -if arenas2.img -of arenapart -bs 8192 -count 97
% dd -if arenas1.img -of arenas -bs 8192 -iseek 97 -count 65536
when the first arena starts after 98 blocks from the beginning of the file?

2012/1/20 David du Colombier <0intro@gmail.com>:
>> that's only 98 blocks of 8192 bytes, not 128 as you mention.
>
> Sorry, I got confused. It's 98 blocks on arena partition and
> 128 blocks on isect partition.
>
> I just tried. This is what I did.
>
> The goal is to manually recopy the first arena from
> the first Venti (arenas1.img) to the second Venti (arenas2.img).
>
> It will work for you as long as you adjust the partition
> names, the number of arenas and the sizes.
>
> # first venti
>
> % dd -if /dev/zero -of arenas1.img -bs 8192 -count 131072
> % dd -if /dev/zero -of isect1.img -bs 8192 -count 16384
> % dd -if /dev/zero -of bloom1.img -bs 8192 -count 8192
> % venti/fmtarenas arenas arenas1.img
> % venti/fmtisect isect isect1.img
> % venti/fmtbloom bloom1.img
> % echo 'index main
>    isect isect1.img
>    arenas arenas1.img
>    bloom bloom1.img' | venti/conf -w arenas1.img
> % venti/fmtindex arenas1.img
> % venti=127.1
> % venti/venti -c arenas1.img -m 20 -h tcp!127.1!8000
> % vac /sys/src/9/pc
> % hget http://127.1:8000/index
> index=main version=1 blocksize=8192 tabsize=524288
>    buckets=16287 div=263706
>    sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
>    arena='arenas0' on arenas1.img at [802816,537673728)
>    version=5 created=1327069595 modified=1327069627
>    written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
>    indexed: clumps=0 compressed clumps=0 data=0 compressed data=0 storage=0
> % venti/sync
> % Kill venti | rc
> % bc
> 802816/8192 = 98
> 537673728/8192 = 65634
> 65634-98 = 65536
>
> # second venti
>
> % dd -if /dev/zero -of arenas2.img -bs 8192 -count 65633
> % dd -if /dev/zero -of isect2.img -bs 8192 -count 16384
> % dd -if /dev/zero -of bloom2.img -bs 8192 -count 8192
> % venti/fmtarenas arenas arenas2.img
> % venti/fmtisect isect isect2.img
> % venti/fmtbloom bloom2.img
> % echo 'index main
>    isect isect2.img
>    arenas arenas2.img
>    bloom bloom2.img' | venti/conf -w arenas2.img
> % venti/fmtindex arenas2.img
> % dd -if arenas2.img -of arenapart -bs 8192 -count 97
> % dd -if arenas1.img -of arenas -bs 8192 -iseek 97 -count 65536
> % cat arenapart arenas > arenas2.img
> % rm -f arenapart arenas
> % venti/buildindex -b arenas2.img
> 0 clumps, 16,287 buckets
> 2012/0120 14:29:29 read index
> venti/buildindex: brand-new index, no work to do
> 2012/0120 14:29:29 arena arenas0: 669 entries
> % venti=127.1
> % venti/venti -c arenas2.img -m 20 -h tcp!127.1!8000
> term% hget http://127.1:8000/index
> index=main version=1 blocksize=8192 tabsize=524288
>    buckets=16287 div=263706
>    sect=isect for buckets [0,16287) buckmax=215 arena=arenas0 at index [1048576,537903104)
>    arena='arenas0' on arenas2.img at [802816,537673728)
>    version=5 created=1327069595 modified=1327069627
>    written: clumps=669 compressed clumps=600 data=4,251,763 compressed data=1,174,341 storage=1,216,488
>    indexed: clumps=0 compressed clumps=0 data=0 compressed data=0
>    storage=0
>
> Honestly, I think you should just use venti/wrarena to write
> backed up arenas to a running Venti. It's much easier.
>
> --
> David du Colombier
>



-- 
Hugo

[9fans] venti backup

[David du Colombier]

[-- Attachment #1: Type: text/plain, Size: 217 bytes --]

This is because each arena have an header (ArenaHead) and a trailer
(ArenaTrail) we would like to copy. The header, in particular, is located
just one block before the start of the arena.

--
David du Colombier

[-- Attachment #2: Type: text/html, Size: 221 bytes --]

Re: [9fans] venti backup

[hugo rivera]

Does the presence of the trailer imply that I should add an extra
block to the arenas backup?
If my last arena is

arena='arenas059' [31676186624,32213057536)

then I should backup 32213057536+8192 bytes instead of 32213057536?

2012/1/20 David du Colombier <0intro@gmail.com>:
> This is because each arena have an header (ArenaHead) and a trailer
> (ArenaTrail) we would like to copy. The header, in particular, is located
> just one block before the start of the arena.
>
> --
> David du Colombier



--
Hugo

Re: [9fans] venti backup

[David du Colombier]

> Does the presence of the trailer imply that I should add an extra
> block to the arenas backup?
> If my last arena is
>
> arena='arenas059' [31676186624,32213057536)
>
> then I should backup 32213057536+8192 bytes instead of 32213057536?

No, the trailer is located at the end of the arena,
just after clump info.

The second number you see is the start of the next arena.
It is preceded by two blocks:

 - the arena header of the next arena,
 - the arena trailer of the precedent arena.

32213057536/8192 = 3932258

In you example, arenas060 start at 3932258, so the header of
arenas060 is 3932257 and the trailer of arenas059 is 3932256.

You should really backup from 1048576-8192 (start of arenas00,
including header) to 32213057536-2*8192 (end of arenas059,
including trailer).

Like many Venti tools, checkarenas check each arena header
and trailer and verify they match. So you should be confident.

--
David du Colombier

Re: [9fans] venti backup

[hugo rivera]

OK, thanks a lot for your help!

2012/1/20 David du Colombier <0intro@gmail.com>:
>> Does the presence of the trailer imply that I should add an extra
>> block to the arenas backup?
>> If my last arena is
>>
>> arena='arenas059' [31676186624,32213057536)
>>
>> then I should backup 32213057536+8192 bytes instead of 32213057536?
>
> No, the trailer is located at the end of the arena,
> just after clump info.
>
> The second number you see is the start of the next arena.
> It is preceded by two blocks:
>
>  - the arena header of the next arena,
>  - the arena trailer of the precedent arena.
>
> 32213057536/8192 = 3932258
>
> In you example, arenas060 start at 3932258, so the header of
> arenas060 is 3932257 and the trailer of arenas059 is 3932256.
>
> You should really backup from 1048576-8192 (start of arenas00,
> including header) to 32213057536-2*8192 (end of arenas059,
> including trailer).
>
> Like many Venti tools, checkarenas check each arena header
> and trailer and verify they match. So you should be confident.
>
> --
> David du Colombier
>



-- 
Hugo