[9fans] Question about fossil

[9fans] Question about fossil

[Riddler]

I'm thinking of turning my raspberry pi into a fossil/venti
combination to store some files and I am curious about how fossil
would handle a particular scenario.

Some background; I have a 1.5TB HDD that I was planning on turning
into a 64GB fossil partition and a ~1472GB venti partition.

It is quite possible the active portion of the filesystem will exceed
64GB. Obviously that is not to say I expect to be changing 64GB of
data between snapshots/archives, but I may want >64GB to be
accessible.

My understanding from reading the wiki and related man pages is that
fossil will allow this because after a file is archived to venti its
blocks on the fossil partition are freed up for re-use and a reference
to the venti data is stored instead.

Onto my question: What if I shrunk that fossil partition to, say, 1GB
and then wrote either more than 1GB in small files or a single 2GB
file.

Will fossil error on the write that pushes it over the edge?
Perhaps 'spill' the extra data over to venti?
Something else entirely?

Re: [9fans] Question about fossil

[Steve Simon]

sadly if fossil overflows it is terminal.

having said this you can refresh your fossil from the last snapshot,
so not everything has gone, but the rule is don't let fossil overflow.

your venti size sounds very big. Perhaps you have a lot of media files,
but remember that duplicate files are merged and take up only one file's
worth of space (de-duplication is done on the files contents not on its name.

also all data is compressed so it takes less space.

-Steve

Re: [9fans] Question about fossil

[Riddler]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

On 8 Jun 2014 00:19, "Steve Simon" <steve@quintile.net> wrote:
>
> sadly if fossil overflows it is terminal.
>
> having said this you can refresh your fossil from the last snapshot,
> so not everything has gone, but the rule is don't let fossil overflow.

Good to know.

> your venti size sounds very big. Perhaps you have a lot of media files,
> but remember that duplicate files are merged and take up only one file's
> worth of space (de-duplication is done on the files contents not on its
name.
>
> also all data is compressed so it takes less space.

I wasn't thinking "I would need a big venti", more  "I only need a small
fossil". My train of thought was because the fossil size is used to store
the unarchived files after which they can be gotten from venti that it
might be practical to only have the fossil be big enough to store the
maximal size of files that will change per day(snapshot interval).

I would struggle to change 16GB a day unless I'm backing up a VM so 64
seemed like it should accommodate any changes and still leave room for lots
of often used files to be kept there (if fossil thinks like that).

I realise there would be a delay as fossil realises it needs to fall back
on venti but I was thinking as they're the same PC and disk the delay would
be negligible.

[-- Attachment #2: Type: text/html, Size: 1587 bytes --]

Re: [9fans] Question about fossil

[Bakul Shah]

On Sat, 07 Jun 2014 21:39:29 BST Riddler <riddler876@gmail.com> wrote:
>
> Onto my question: What if I shrunk that fossil partition to, say, 1GB
> and then wrote either more than 1GB in small files or a single 2GB
> file.

Why would you want to make the fossil partition that small?

I would keep it at least twice as large as the largest file
I'd ever want to create.

> Will fossil error on the write that pushes it over the edge?
> Perhaps 'spill' the extra data over to venti?
> Something else entirely?

I haven't studied fossil but AFAIK it won't spill data to
venti when it runs low on disk. Rather, you set it up to take
daily snapshots so the partition should be large enough to
hold all the new data you may generate in a day.

Other things to note:

- make sure you mirror or backup the venti disk or else you
  may lose the only copy of a block!
- try this out on a small scale before you commit to it, as I
  suspect you'll run into various limits and may be bugs. Do
  report what you discover.
- performance will likely be poor. For better performance you
  may want to keep venti index on a separate (flash) disk.
- it would be nice if you can come up with a workable setup
  for a venti server!

Re: [9fans] Question about fossil

[erik quanstrom]

> - try this out on a small scale before you commit to it, as I
>   suspect you'll run into various limits and may be bugs. Do
>   report what you discover.
> - performance will likely be poor. For better performance you
>   may want to keep venti index on a separate (flash) disk.
> - it would be nice if you can come up with a workable setup
>   for a venti server!

usb performance is ~4-7MB/s.  this is the best you can hope for
from the disk.  venti will only slow this down by multiplying
disk accesses and being a bit seeky.  keep in mind if you're
using this for a venti server, that usb b/w needs to be shared
with tcp.

- erik

Re: [9fans] Question about fossil

[erik quanstrom]

> I wasn't thinking "I would need a big venti", more  "I only need a small
> fossil". My train of thought was because the fossil size is used to store
> the unarchived files after which they can be gotten from venti that it
> might be practical to only have the fossil be big enough to store the
> maximal size of files that will change per day(snapshot interval).
>
> I would struggle to change 16GB a day unless I'm backing up a VM so 64
> seemed like it should accommodate any changes and still leave room for lots
> of often used files to be kept there (if fossil thinks like that).

why bother optimizing this?  fossil is going to be <1% of the disk even
if you make it silly huge.

- erik

Re: [9fans] Question about fossil

[Steve Simon]

> I wasn't thinking "I would need a big venti", more  "I only need a small
> fossil". My train of thought was because the fossil size is used to store
> the unarchived files after which they can be gotten from venti that it
> might be practical to only have the fossil be big enough to store the
> maximal size of files that will change per day(snapshot interval).
>

Quite right, 16Gb is fine.

> I would struggle to change 16GB a day unless I'm backing up a VM so 64
> seemed like it should accommodate any changes and still leave room for lots
> of often used files to be kept there (if fossil thinks like that).

No fossil does not do this, after the snapshot fossil is emptied completely
and a pointer is placed at the root directory of fossil pointing to venti
so all access now go to venti. From then on fossil runs as copy on write
so it stores changes since the last snapshot.

> I realise there would be a delay as fossil realises it needs to fall back
> on venti but I was thinking as they're the same PC and disk the delay would
> be negligible.

There should be no delay in fossil as such.

I will admit venti is not as fast as a more traditional fs, or even cwfs/ken's fs
on plan9 but it has its own advantages which outweigh this for me.

BTW: I have a mirror of 2 disks for fossil and venti and plan to add a SSD
to this mirror in the hope that this will improve venti performance.

It is all a changing of thinking - for example, never truncate logfiles,
as truncating them actually uses more space in venti than just letting them grow.

never worry about cloneing large directories, its (almost) free.

if you have big files you don't want to keep in venti use chmod -t on them
to stop archiving (keep the file in fossil only). This means they are not backed up
in venti but I find it helpful for things like downloaded ISO files where they
can be easily regenerated.

-Steve

Re: [9fans] Question about fossil

[erik quanstrom]

> It is all a changing of thinking - for example, never truncate logfiles,
> as truncating them actually uses more space in venti than just letting them grow.
>
> never worry about cloneing large directories, its (almost) free.

in my mind these are not related to the content-addressed storage feature
of venti, but rather the idea that storage is practically infinite.

> if you have big files you don't want to keep in venti use chmod -t on them
> to stop archiving (keep the file in fossil only). This means they are not backed up
> in venti but I find it helpful for things like downloaded ISO files where they
> can be easily regenerated.

keeping the fact that storage is infinte in mind, i don't understand +t.
it breaks the model, without a big benefit.  the n/other model announces
itself as non-archived storage, and won't be lost if fossil is recovered from
archive.

- erik

Re: [9fans] Question about fossil

[Bakul Shah]

On Sun, 08 Jun 2014 03:56:24 EDT erik quanstrom <quanstro@quanstro.net> wrote:
> > - try this out on a small scale before you commit to it, as I
> >   suspect you'll run into various limits and may be bugs. Do
> >   report what you discover.
> > - performance will likely be poor. For better performance you
> >   may want to keep venti index on a separate (flash) disk.
> > - it would be nice if you can come up with a workable setup
> >   for a venti server!
>
> usb performance is ~4-7MB/s.  this is the best you can hope for
> from the disk.  venti will only slow this down by multiplying
> disk accesses and being a bit seeky.  keep in mind if you're
> using this for a venti server, that usb b/w needs to be shared
> with tcp.

The last time I measured this (Aug 2012) raw disk write was
10MB/s, file writes were 2MB/s. On the same h/w & disk linux
got 25MB/s (don't recall file throughput). And Linux gets
11.3MB/s ethernet throughput compared 3.7MB/s on 9pi (both
with ttcp). Linux tcp throughput is close to linespeed.

Might almost be worth using p9p vent under linux!  And keep
isects on the sdcard and arenas on an external disk.

Re: [9fans] Question about fossil

[erik quanstrom]

> The last time I measured this (Aug 2012) raw disk write was
> 10MB/s, file writes were 2MB/s. On the same h/w & disk linux
> got 25MB/s (don't recall file throughput). And Linux gets
> 11.3MB/s ethernet throughput compared 3.7MB/s on 9pi (both
> with ttcp). Linux tcp throughput is close to linespeed.

i can push the usb hard enough to trigger bugs.  :-)
i usually get about 6MB/s.

there's only one trick: use qmalloc instead of pool.
pool astounds with its slowness.  the code is in 9atom.

- erik

Re: [9fans] Question about fossil

[Riddler]

> Why would you want to make the fossil partition that small?
>
> I would keep it at least twice as large as the largest file
> I'd ever want to create.

I was not actually planning on making it that small, I was just
curious as to how fossil would react.
It was brought about mainly because the wiki states that sources only
uses ~512MB for fossil.
Which is what got me thinking "well what if it gets full?"


> Other things to note:
>
> - make sure you mirror or backup the venti disk or else you
>   may lose the only copy of a block!
> - try this out on a small scale before you commit to it, as I
>   suspect you'll run into various limits and may be bugs. Do
>   report what you discover.
> - performance will likely be poor. For better performance you
>   may want to keep venti index on a separate (flash) disk.
> - it would be nice if you can come up with a workable setup
>   for a venti server!

It's just going on my raspberry pi that I can spare (which is why I got it).
The disk will be fully backed up, so I'm not worried about it failing
for the moment.

It will be a week or two before I get the chance to give it a go, but
I will certainly report how it goes and what bugs or limits (if any) I
trip over.

Re: [9fans] Question about fossil

[Riddler]

> why bother optimizing this?  fossil is going to be <1% of the disk even
> if you make it silly huge.

64GB was just a number that seemed like it should accommodate pretty
much anything I could need.
As Steve points out fossil won't do any caching of often used files so
I'll probably cut it in half to 32GB see how it goes.

Re: [9fans] Question about fossil

[Lyndon Nerenberg]

[-- Attachment #1: Type: text/plain, Size: 213 bytes --]


On Jun 9, 2014, at 1:21 PM, Riddler <riddler876@gmail.com> wrote:

> It was brought about mainly because the wiki states that sources only
> uses ~512MB for fossil.

I suspect that's wildly out of date.


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 817 bytes --]

Re: [9fans] Question about fossil

[erik quanstrom]

On Mon Jun  9 17:13:09 EDT 2014, lyndon@orthanc.ca wrote:

>
> On Jun 9, 2014, at 1:21 PM, Riddler <riddler876@gmail.com> wrote:
>
> > It was brought about mainly because the wiki states that sources only
> > uses ~512MB for fossil.
>
> I suspect that's wildly out of date.

a basic install requires about 512mb, which all must be in fossil at
the same time.  so that's correct as far as it goes.

nontheless, i think that 2% or a minimum of 1G and a maximum
of about 20G makes sense for fossil.  i have run various file servers
out of space, and it's not pretty.

- erik

Re: [9fans] Question about fossil

[Bakul Shah]

On Mon, 09 Jun 2014 17:25:51 EDT erik quanstrom <quanstro@quanstro.net> wrote:
> On Mon Jun  9 17:13:09 EDT 2014, lyndon@orthanc.ca wrote:
>
> >
> > On Jun 9, 2014, at 1:21 PM, Riddler <riddler876@gmail.com> wrote:
> >
> > > It was brought about mainly because the wiki states that sources only
> > > uses ~512MB for fossil.
> >
> > I suspect that's wildly out of date.
>
> a basic install requires about 512mb, which all must be in fossil at
> the same time.  so that's correct as far as it goes.
>
> nontheless, i think that 2% or a minimum of 1G and a maximum
> of about 20G makes sense for fossil.  i have run various file servers
> out of space, and it's not pretty.

Over the weekend I was playing with fossil and "copied" my
fossil partition using its last score, swapped the two disks
(under virtualbox) and rebooted.  df now shows 1MB in use! So
if you init fossil from the score of an existing installation,
you can make do with a lot less space -- only depends on how
much new stuff you create every day! Even there you can
probably write a script that watches df and when it reaches
some limit creates an archival snapshot or just snapshot
every hour or so!

This idea can drastically reduce new installation time.
Someone (vsrinivas?) has created vtrc.c, a venti proxy, a
variation of Russ's venti/ro.c. This can probably be enhanced
so that if a block is not found in your local venti, it asks a
public venti (and writes back that block to the local venti).

Re: [9fans] Question about fossil

[erik quanstrom]

> Over the weekend I was playing with fossil and "copied" my
> fossil partition using its last score, swapped the two disks
> (under virtualbox) and rebooted.  df now shows 1MB in use! So
> if you init fossil from the score of an existing installation,
> you can make do with a lot less space -- only depends on how
> much new stuff you create every day! Even there you can
> probably write a script that watches df and when it reaches
> some limit creates an archival snapshot or just snapshot
> every hour or so!

i am not sure i follow along.  to me, 1g of disk space is a trival
amount, and  in cases where space might be a bit tight, like on
a sd card, i would think reliablity would push one to put venti
on another media.

> This idea can drastically reduce new installation time.
> Someone (vsrinivas?) has created vtrc.c, a venti proxy, a
> variation of Russ's venti/ro.c. This can probably be enhanced
> so that if a block is not found in your local venti, it asks a
> public venti (and writes back that block to the local venti).

isn't this is trading a one-time small cost for a ongoing cost, and
a set of new problems.  what if the public venti is out-of-sync, or
gone?

- erik

Re: [9fans] Question about fossil

[Bakul Shah]

On Mon, 09 Jun 2014 18:22:26 EDT erik quanstrom <quanstro@quanstro.net> wrote:
> > Over the weekend I was playing with fossil and "copied" my
> > fossil partition using its last score, swapped the two disks
> > (under virtualbox) and rebooted.  df now shows 1MB in use! So
> > if you init fossil from the score of an existing installation,
> > you can make do with a lot less space -- only depends on how
> > much new stuff you create every day! Even there you can
> > probably write a script that watches df and when it reaches
> > some limit creates an archival snapshot or just snapshot
> > every hour or so!
>
> i am not sure i follow along.  to me, 1g of disk space is a trival
> amount, and  in cases where space might be a bit tight, like on
> a sd card, i would think reliablity would push one to put venti
> on another media.

Just pointing out that 512MB or whatever is the new
requirement only applies to a fresh install.

> > This idea can drastically reduce new installation time.
> > Someone (vsrinivas?) has created vtrc.c, a venti proxy, a
> > variation of Russ's venti/ro.c. This can probably be enhanced
> > so that if a block is not found in your local venti, it asks a
> > public venti (and writes back that block to the local venti).
>
> isn't this is trading a one-time small cost for a ongoing cost, and
> a set of new problems.  what if the public venti is out-of-sync, or
> gone?

Just pointing out that we haven't explored venti enough!
Other solutions are possible (with their own problems).

With the trick I am talking about, there is nothing to stop
you from connecting to N different remote ventis.  In effect
your local (by that I mean under your control, not necessarily
on the same machine) venti can be treated as just a buffer!

In spirit this is like much like what bittorrent does but much
more flexible.  Like in bittorrent you pull all the stuff you
depend on (but that can be done in background). You can even
start streaming your remotely stored video right away!

Re: [9fans] Question about fossil

[Brian L. Stuart]

> With the trick I am talking about, there is nothing to stop
> you from connecting to N different remote ventis.  In effect
> your local (by that I mean under your control, not necessarily
> on the same machine) venti can be treated as just a buffer!
 
I took a look at some things along those lines a few years
back in the context of file systems aimed at laptops where
the buffer allowed for operation when disconnected from
the main server.  It wasn't strictly tied to venti as a back-end.
There's a paper on it in the proceedings from IWP9 '09.

http://4e.iwp9.org/papers/lapfs.pdf

It would be interesting to build the same sort of thing in
the fossil-venti connection rather than in the 9P path.

BLS

Re: [9fans] Question about fossil

[Bakul Shah]

On Mon, 09 Jun 2014 17:15:39 PDT "Brian L. Stuart" <blstuart@bellsouth.net> wrote:
> > With the trick I am talking about, there is nothing to stop
> > you from connecting to N different remote ventis.=A0 In effect
> > your local (by that I mean under your control, not necessarily
> > on the same machine) venti can be treated as just a buffer!
> =20
> I took a look at some things along those lines a few years
> back in the context of file systems aimed at laptops where
> the buffer allowed for operation when disconnected from
> the main server.  It wasn't strictly tied to venti as a back-end.
> There's a paper on it in the proceedings from IWP9 '09.
>
> http://4e.iwp9.org/papers/lapfs.pdf
>
> It would be interesting to build the same sort of thing in
> the fossil-venti connection rather than in the 9P path.

Interesting....  Wouldn't you get roughly the same semantics
(modulo symlinks) with fossil +  a local venti proxy + venti?

(Answering myself) Not really since in your scheme the
fileserver copy can change independent of your local copy and
venti can't deal with that as it implements the "many worlds
interpretation"! Even if A & B start with fossil initialized
to the same score, their further histories would diverge. We
need something more.

Thanks for the reference.

Re: [9fans] Question about fossil

[Nicolas Bercher]

On 08/06/2014 18:56, Bakul Shah wrote:
> On Sun, 08 Jun 2014 03:56:24 EDT erik quanstrom <quanstro@quanstro.net> wrote:
>>> - try this out on a small scale before you commit to it, as I
>>>    suspect you'll run into various limits and may be bugs. Do
>>>    report what you discover.
>>> - performance will likely be poor. For better performance you
>>>    may want to keep venti index on a separate (flash) disk.
>>> - it would be nice if you can come up with a workable setup
>>>    for a venti server!
>>
>> usb performance is ~4-7MB/s.  this is the best you can hope for
>> from the disk.  venti will only slow this down by multiplying
>> disk accesses and being a bit seeky.  keep in mind if you're
>> using this for a venti server, that usb b/w needs to be shared
>> with tcp.
>
> The last time I measured this (Aug 2012) raw disk write was
> 10MB/s, file writes were 2MB/s. On the same h/w & disk linux
> got 25MB/s (don't recall file throughput). And Linux gets
> 11.3MB/s ethernet throughput compared 3.7MB/s on 9pi (both
> with ttcp). Linux tcp throughput is close to linespeed.

The R-Pi gave me poor results when copying files from the network to a
usb disk (something around 4MB/s).  As Erik mentioned it, all is usb...
I think the R-Pi is not a valuable device for data storage (Linux or
Plan 9).  I'd prefer to use it as a 24/7 terminal, given its low power
consumption.

Nicolas