[9fans] fossil woe

[9fans] fossil woe

[Bruce Ellis]

I was copying a chunk of data from one machine to another
(both fossil+venti).  The target machine has a huge venti store.

Don't try this at home.  The fossil "filled up" and the machine
died badly, wouldn't even reboot.  So fossil apparently is not a cache.

cache.c:698:  * BUG: if the disk is full, should we flush some of it to Venti?

Now I've "rescued" the machine by booting from floppy, creating
partitions the same size as fossil and arena, copying them and
reinstalling.

Things groove along again, as long as I don't consider fossil to
be a cache, but I'd like some data from the saved arena
(screw the cached stuff).  So all I need is a score.  I seem to
recall methods of scrounging for scores but can't find anything
in my searches.  I would appreciate any help.

brucee

Re: [9fans] fossil woe

[Russ Cox]

> cache.c:698:  * BUG: if the disk is full, should we flush some of it to Venti?

Should we?  It's still not clear to me.  There are fossil console
commands you could have typed to free up a little space
and run an archive to venti.

To find your old vac scores, see /sys/src/cmd/venti/dumpvacroots.

Russ

Re: [9fans] fossil woe

[Eric Van Hensbergen]

On 12/6/05, Russ Cox <rsc@swtch.com> wrote:
> > cache.c:698:  * BUG: if the disk is full, should we flush some of it to Venti?
>
> Should we?  It's still not clear to me.  There are fossil console
> commands you could have typed to free up a little space
> and run an archive to venti.
>

I could see value in treating fossil as a cache.  We're looking at a
bunch of opportunities both for cacheing wide-area-storage (aka WAFS
style stuff) as well as local-disk cache of cluster storage (so you
can use your SAN/venti for backup, but still have quick local access
to the stuff you use the most).

        -eric

Re: [9fans] fossil woe

[Bruce Ellis]

of course. or don't call it a cache but "something that may
fill up and screw you over".  the machine was far too hosed
to connect to fscons and in any case telling a user "if you
do too much stuff you will have to call me to fix fossil" is
not aacceptable.

brucee

On 12/7/05, Russ Cox <rsc@swtch.com> wrote:
> > cache.c:698:  * BUG: if the disk is full, should we flush some of it to Venti?
>
> Should we?  It's still not clear to me.  There are fossil console
> commands you could have typed to free up a little space
> and run an archive to venti.
>
> To find your old vac scores, see /sys/src/cmd/venti/dumpvacroots.
>
> Russ

Re: [9fans] fossil woe

[Russ Cox]

> of course. or don't call it a cache but "something that may
> fill up and screw you over".  the machine was far too hosed
> to connect to fscons and in any case telling a user "if you
> do too much stuff you will have to call me to fix fossil" is
> not aacceptable.

I'm not claiming, under any circumstances, that fossil
becoming unusable is acceptable.  However, the solution
is not as simple as "just write the blocks to venti".
First, writing blocks to venti requires some knowledge
of the rest of the file server state, and you can't always
write any block to venti.  Second, some people don't even
use venti and they can fill their fossils too.

There are plenty of subtle things going on in fossil that make
it much harder to handle the disk being full than in a conventional
file system.

Fossil needs some notion of being close to full, at which point
its behavior would change.  Automatic snapshots would stop,
as would mtime updates and any other source of automatic
dirtying of blocks.  There should be a few blocks reserved for
snapshots taken in response to manual console commands.
Then when fossil fills, you can remove recently written data
(if some runaway program has filled the disk) or connect to
the console and delete old snapshots or take an archival
snapshot.

If someone wants to work on making this happen, I would be
happy to provide further details.

Russ

Re: [9fans] fossil woe

[Bruce Ellis]

so it should be made clear that fossil is not appropriate for
large files or file system traffic.  if i wish to create a 5G file
and fossil is 10G can i?  well it depends on what has happened
since the last snap.

brucee

On 12/7/05, Russ Cox <rsc@swtch.com> wrote:
> > of course. or don't call it a cache but "something that may
> > fill up and screw you over".  the machine was far too hosed
> > to connect to fscons and in any case telling a user "if you
> > do too much stuff you will have to call me to fix fossil" is
> > not aacceptable.
>
> I'm not claiming, under any circumstances, that fossil
> becoming unusable is acceptable.  However, the solution
> is not as simple as "just write the blocks to venti".
> First, writing blocks to venti requires some knowledge
> of the rest of the file server state, and you can't always
> write any block to venti.  Second, some people don't even
> use venti and they can fill their fossils too.
>
> There are plenty of subtle things going on in fossil that make
> it much harder to handle the disk being full than in a conventional
> file system.
>
> Fossil needs some notion of being close to full, at which point
> its behavior would change.  Automatic snapshots would stop,
> as would mtime updates and any other source of automatic
> dirtying of blocks.  There should be a few blocks reserved for
> snapshots taken in response to manual console commands.
> Then when fossil fills, you can remove recently written data
> (if some runaway program has filled the disk) or connect to
> the console and delete old snapshots or take an archival
> snapshot.
>
> If someone wants to work on making this happen, I would be
> happy to provide further details.
>
> Russ
>

Re: [9fans] fossil woe

[rog]

> Fossil needs some notion of being close to full, at which point
> its behavior would change.  Automatic snapshots would stop,
> as would mtime updates and any other source of automatic
> dirtying of blocks.  There should be a few blocks reserved for
> snapshots taken in response to manual console commands.
> Then when fossil fills, you can remove recently written data
> (if some runaway program has filled the disk) or connect to
> the console and delete old snapshots or take an archival
> snapshot.

this seems fine, but... it still seems somewhat arduous when all
one really wants to do is copy a load of data to venti.

one idea:

why not make it possible to directly insert vac archives into
a fossil filesystem?

e.g. with a new fossilcons command:

	createvac path score uid gid perm

fossil could remap the (possibly spurious) usernames in the
vac archive, for instance by mapping all user and group ids to
the given uid and gid.

then a large quantity of new data could be "vac"ed directly
onto venti, and then made available within fossil.

in combination with the "vac" console command, this could make
other previously hard things straightforward; for example replicating
or moving huge directories from place to place without
copying all the data.

would this be very hard within the current fossil structure?

Re: [9fans] fossil woe

[Steve Simon]

> why not make it possible to directly insert vac archives into
> a fossil filesystem?

I wanted to do this, I even got some hints from russ, but in
the end I failed - the pain/gain ratio of understanding that much
about fossil seemed too big.

It still sits on my to-do list though.

-Steve

Re: [9fans] fossil woe

[Bruce Ellis]

this misses the point.  a cache must cope with an "I'm full"
condition otherwise it can and will fail.  i'd much rather a
long delay on a file-op when fossil does a "I'm nearly full"
snap than a few days to rescue a smashed system.

brucee

On 12/8/05, rog@vitanuova.com <rog@vitanuova.com> wrote:
> > Fossil needs some notion of being close to full, at which point
> > its behavior would change.  Automatic snapshots would stop,
> > as would mtime updates and any other source of automatic
> > dirtying of blocks.  There should be a few blocks reserved for
> > snapshots taken in response to manual console commands.
> > Then when fossil fills, you can remove recently written data
> > (if some runaway program has filled the disk) or connect to
> > the console and delete old snapshots or take an archival
> > snapshot.
>
> this seems fine, but... it still seems somewhat arduous when all
> one really wants to do is copy a load of data to venti.
>
> one idea:
>
> why not make it possible to directly insert vac archives into
> a fossil filesystem?
>
> e.g. with a new fossilcons command:
>
>        createvac path score uid gid perm
>
> fossil could remap the (possibly spurious) usernames in the
> vac archive, for instance by mapping all user and group ids to
> the given uid and gid.
>
> then a large quantity of new data could be "vac"ed directly
> onto venti, and then made available within fossil.
>
> in combination with the "vac" console command, this could make
> other previously hard things straightforward; for example replicating
> or moving huge directories from place to place without
> copying all the data.
>
> would this be very hard within the current fossil structure?
>
>

Re: [9fans] fossil woe

[Russ Cox]

> this misses the point.  a cache must cope with an "I'm full"
> condition otherwise it can and will fail.  i'd much rather a
> long delay on a file-op when fossil does a "I'm nearly full"
> snap than a few days to rescue a smashed system.

you have the source.  please fix it.

russ

Re: [9fans] fossil woe

[Bruce Ellis]

I don't have this problem on Inferno because my file system
cache (which delegates to venti) doesn't have this bug.

I didn't realize that anyone would write a cache that breaks
when it fills.  I'm not touching fossil.

brucee

On 12/8/05, Russ Cox <rsc@swtch.com> wrote:
> > this misses the point.  a cache must cope with an "I'm full"
> > condition otherwise it can and will fail.  i'd much rather a
> > long delay on a file-op when fossil does a "I'm nearly full"
> > snap than a few days to rescue a smashed system.
>
> you have the source.  please fix it.
>
> russ

Re: [9fans] fossil woe

[jmk]

On Wed Dec  7 16:55:38 EST 2005, bruce.ellis@gmail.com wrote:
> I don't have this problem on Inferno because my file system
> cache (which delegates to venti) doesn't have this bug.
> 
> I didn't realize that anyone would write a cache that breaks
> when it fills.  I'm not touching fossil.
> 
> brucee

How I wish that none of us ever had to.

Re: [9fans] fossil woe

[Russ Cox]

If I had the time, which I don't, there are many things
I would fix about fossil.  Behavior when full is high on
the list, as is the ability to specify a score in a create
command.  Fossil, in contrast to most Plan 9 programs,
was a collaborative effort, which meant that no one
understood the whole picture for too long.

If anyone does have time and wants to work on these
problems, I am always happy to provide guidance.

Russ

Re: [9fans] fossil woe

[jmk]

On Wed Dec  7 16:15:27 EST 2005, bruce.ellis@gmail.com wrote:
> this misses the point.  a cache must cope with an "I'm full"
> condition otherwise it can and will fail.  i'd much rather a
> long delay on a file-op when fossil does a "I'm nearly full"
> snap than a few days to rescue a smashed system.
> 
> brucee

     FOSSIL(4)                                               FOSSIL(4)
	 ...
     BUGS
          It is likely that the disk format (but not the Venti format)
          will change in the future, to make the disk a full cache
          rather than just a write buffer.  Changing to the new format
          will require reformatting the disk as in the example above,
          but note that this will preserve most of the file system
          (all but /snapshot) with little effort.

Re: [9fans] fossil woe

[geoff]

As a point of comparison, if the mag disk `cache' in ken fs filled
with dirty blocks, the file server would be unable to accept any more
dirty blocks and would not perform an automatic dump to empty the
cache.  Ken told me that he'd looked at triggering automatic dumps
when the cache got nearly full but that it didn't look
straightforward.