[9fans] fossil+plan9: disk full revisited

[9fans] fossil+plan9: disk full revisited

[Georg Lehner]

Hello!

It seems that this year I am the one who stumbled over the infamous
"cacheAllocBlock: xxx1 disk is full" situation.  No problem: I
formatted and started over :-0


The setup is a 100GB plan9 partition with about 14GB fossil and the
rest venti, set up for confidence and performance tests of
Plan9/fossil+venti in order to evaluate it for production use as a
fileserver in the internal network at my work, with some Linux and a
lot of Windows users.

For a start, I copied a disk image with about 18GB from a Linux server
with u9fs to the Plan9 file server, which obviously choked fossil.

Two questions:

- Would it help to write blocks to venti and free them on fossil, if a
  snapshot is taken in the middle of a transfer of one file bigger
  than the whole fossil partition?

- How much would fossil have to be altered, to make it automatically
  write blocks to venti when only a certain amount of blocks is free
  anymore.

While timed snapshots are really a nice thing for archiving, this
feature maybe would make fossil+venti practically maintenance free.

Regards,

        Jorge-León

Re: [9fans] fossil+plan9: disk full revisited

[Robert Raschke]

Jorge-León wrote:
> It seems that this year I am the one who stumbled over the infamous
> "cacheAllocBlock: xxx1 disk is full" situation.  No problem: I
> formatted and started over :-0
> 
> 
> The setup is a 100GB plan9 partition with about 14GB fossil and the
> rest venti, set up for confidence and performance tests of
> Plan9/fossil+venti in order to evaluate it for production use as a
> fileserver in the internal network at my work, with some Linux and a
> lot of Windows users.
> 
[...snip...]
> 
> While timed snapshots are really a nice thing for archiving, this
> feature maybe would make fossil+venti practically maintenance free.

You have to make sure that your fossil snapshots are recycled.

When you set up aotumatic snapshotting to venti using the fossil
snaptime command, make sure to set all three times: the archive to
venti, create fossil snapshot, and how long to keep fossil snapshots
around.  My fossil is set up with

snaptime -a 0200 -s 60 -t 7200

Archive to venti at 2AM, keep hourly fossil snapshots and throw away
fossil snapshts oder than 7200 minutes (5 days).

Without the last entry, your fossil will _definitely_ run out of space.

That has happened to me twice in the past couple of years.  I always
manage to get out of it by booting from a CD, mounting my fossil by
hand and running the 'epoch' command to ditch accumulated junk.  I've so
far never had to rebuild my fossil/venti from scratch.  That's going
back about two and half years now.

Robby

Re: [9fans] fossil+plan9: disk full revisited

[Dave Eckhardt]

My whimsical suggestion is that writing a file larger than
fossil's write buffer should be like approaching the event
horizon of a black hole:  fossil could deliberately delay
responses to Twrite's (a) generally as space gets low, and
(b) specifically for people extending large files.

Dave Eckhardt

Re: [9fans] fossil+plan9: disk full revisited

[Georg Lehner]

I did it again!

on a second machine I tried

 dd -if=/dev/null -of=fillit.all bs=1024k count=2048

with a 1.8G fossil partition to be sure to fill it up completely.

I learned four things from it:

- fossil does not compress duplicated data blocks
- fossil cannot store total data bigger then its partition size
- read the documentation
- Read the Documentation!

The excelent paper:

  http://www.cs.bell-labs.com/sys/doc/fossil.pdf

tells it all, it's a pity it is not referenced in the papers section
on Bell labs Plan9 page.  You must instead go to the wiki and there to
the papers section.

I think the Plan9 papers section tend to confuse, since the "old" file
server is still referenced there although it seems more of historical
interest then as introductory material.

On the other hand my perception was biased by the installers
suggestion of the distribution of space between fossil and venti.

I have a 12GByte Harddisk, which was divided into:

  fossil    1.84G
  arenas    9.24G

considering that the filesystem gets compressed in venti, after some
thought I'd use as much fossil as possible and the smallest venti
possible for a reasonable archival time span (1 year?).  If venti runs
out of space one would archive some arenas to CD, DVD etc.

With a dumb estimate of 30% compression ratio one would give about 60%
of a disk to fossil, and 40% to venti.  What are venti compression
ratios experienced in real live?!


However I am not sure, if my conclusions are right. At

  http://www.magma.com.ni/moin/Plan9Tutorial/FossilVenti

my current understandings about fossil+venti are put together. I would
be very pleased to get feedback and corrections about it.

Regards


    Jorge-León

Re: [9fans] fossil+plan9: disk full revisited

[geoff]

Fossil does not eliminate duplicate blocks nor compress blocks, but
venti does both, and it is intended that serious use of fossil should
be backed with venti (which arenas should be stored on a RAID or
equivalent).  Using fossil without a backing venti gives the
equivalent of an `other' partition on Ken's file server: scratch
space.

The fossil write buffer needs to be only as large as the largest set
of blocks dirtied between dumps to venti, loosely speaking, so it can
typically be much smaller than the venti arenas backing it.  Edith,
our current primary file server, has 400GB of venti arenas, with
another 100GB available but not yet configured, and 34GB of fossil
write buffer, of which df reports 3% used.  87% of the arenas are
used.  (Yes, we have a new, larger file server ready to replace
edith.)