[9fans] fossil: disk is full

[9fans] fossil: disk is full

[Kenji Arisawa]

Hello,

I have some times observed message "disk is full" from fossil server
when I copied large amount of data to fossil.
The problem is in that:
I could not execute fossilcons command "snap -a" to sweep the disk
data to venti. I guess that "snap -a" requires some working space
in the disk.
If my guess is true, I think it is better fossil protects the working 
space
from being filled by file data.

Kenji Arisawa

Re: [9fans] fossil: disk is full

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 109 bytes --]

You can do what rsc said time ago, delete some snapshots before
using snap -a. You can use epoch to do so.

[-- Attachment #2: Type: message/rfc822, Size: 1709 bytes --]

From: Kenji Arisawa <arisawa@ar.aichi-u.ac.jp>
To: 9fans@cse.psu.edu
Subject: [9fans] fossil: disk is full
Date: Fri, 7 Mar 2003 12:48:56 +0900
Message-ID: <B8E6C0E6-504F-11D7-8202-000393A941BC@ar.aichi-u.ac.jp>

Hello,

I have some times observed message "disk is full" from fossil server
when I copied large amount of data to fossil.
The problem is in that:
I could not execute fossilcons command "snap -a" to sweep the disk
data to venti. I guess that "snap -a" requires some working space
in the disk.
If my guess is true, I think it is better fossil protects the working 
space
from being filled by file data.

Kenji Arisawa

Re: [9fans] fossil: disk is full

[Lucio De Re]

On Fri, Mar 07, 2003 at 08:52:05AM +0100, Fco.J.Ballesteros wrote:
> 
> You can do what rsc said time ago, delete some snapshots before
> using snap -a. You can use epoch to do so.

Reserving space is ugly, but comes closer to the principle of least
surprise.  Is there a (fixed) amount of disk space worth reserving?

++L

Re: [9fans] fossil: disk is full

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 252 bytes --]

I try just to keep the assumption that disk space is
unlimited. If you run fossil backed up by venti, a script
that clears all but the last N snapshots can keep your
fossil partition with enough space that it's not worth
reserving anything, IMHO.

[-- Attachment #2: Type: message/rfc822, Size: 2174 bytes --]

From: Lucio De Re <lucio@proxima.alt.za>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] fossil: disk is full
Date: Fri, 7 Mar 2003 10:17:11 +0200
Message-ID: <20030307101711.F13434@cackle.proxima.alt.za>

On Fri, Mar 07, 2003 at 08:52:05AM +0100, Fco.J.Ballesteros wrote:
> 
> You can do what rsc said time ago, delete some snapshots before
> using snap -a. You can use epoch to do so.

Reserving space is ugly, but comes closer to the principle of least
surprise.  Is there a (fixed) amount of disk space worth reserving?

++L

Re: [9fans] fossil: disk is full

[Kenji Arisawa]

Hello,

> You can do what rsc said time ago, delete some snapshots before
> using snap -a. You can use epoch to do so.
I did that, but what happen if I have snapshot?

Kenji Arisawa

Re: [9fans] fossil: disk is full

[Kenji Arisawa]

Sorry
 >I did that, but what happen if I have snapshot?
I did that, but what happen if I have no snapshot?

Kenji Arisawa

Re: [9fans] fossil: disk is full

[Lucio De Re]

On Fri, Mar 07, 2003 at 05:43:55PM +0900, Kenji Arisawa wrote:
> 
> Sorry
>  >I did that, but what happen if I have snapshot?
> I did that, but what happen if I have no snapshot?
> 
Well, if Nemo's right, you'd best start making snapshots and,
unavoidably, get more disk space :-)

++L

Re: [9fans] fossil: disk is full

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 182 bytes --]

That sounds funny to me. I think we have a fortune...

"If you need to prevent problems due to a disk full, you'd better start
keeping snapshots on it."

Sorry, couldn't resist.

[-- Attachment #2: Type: message/rfc822, Size: 2137 bytes --]

From: Lucio De Re <lucio@proxima.alt.za>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] fossil: disk is full
Date: Fri, 7 Mar 2003 10:59:58 +0200
Message-ID: <20030307105958.H13434@cackle.proxima.alt.za>

On Fri, Mar 07, 2003 at 05:43:55PM +0900, Kenji Arisawa wrote:
> 
> Sorry
>  >I did that, but what happen if I have snapshot?
> I did that, but what happen if I have no snapshot?
> 
Well, if Nemo's right, you'd best start making snapshots and,
unavoidably, get more disk space :-)

++L

Re: [9fans] fossil: disk is full

[Lucio De Re]

On Fri, Mar 07, 2003 at 10:11:09AM +0100, Fco.J.Ballesteros wrote:
> 
> That sounds funny to me. I think we have a fortune...
> 
> "If you need to prevent problems due to a disk full, you'd better start
> keeping snapshots on it."
> 
You said it!

++L

Re: [9fans] fossil: disk is full

[Axel Belinfante]

Just to check my understanding:
Suppose I want to copy stuff to fossil (I guess the preferred way
would be using mkfs/mkext?), that's much much more than the fossil
write buffer, can I get away with it by often enough doing 'snap -a'?
(maybe by automating the writing of 'snap -a' to the fossile console
 while the copy runs in parallel?)

(I was about to suggest: setting automatic archival snapshots
 with snaptime -a such that they happen often enough,
 but I saw that an automatic archival snapshot can be only
 specified once a day)

Background: I want to copy a collection of mp3's from FAT32
partitions to a fossil server I'm still setting up.
Each individual FAT32 partition contains more than the fossil
write buffer. Quite a number of mp3's appear (at least) twice
in the directory hierarchy, so I thought it would make sense
to store them on the venti archived part instead of on a 'once'
partition, to share duplicates.


Hmmm... or would it be much simpler to just vac the dos files?

Axel.

> > I did that, but what happen if I have no snapshot?
> > 
> Well, if Nemo's right, you'd best start making snapshots and,
> unavoidably, get more disk space :-)

Re: [9fans] fossil: disk is full

[Russ Cox]

Yes, you can get away with snap -a occasionally
during the write buffer, perhaps combined with
stop mkfs |rc and restarting it.

Re: [9fans] fossil: disk is full

[Fco.J.Ballesteros]

[-- Attachment #1: Type: text/plain, Size: 123 bytes --]

If you have no snapshot you can always sacrifice
a file. (for example, one that you can later
generate from other files).

[-- Attachment #2: Type: message/rfc822, Size: 1481 bytes --]

From: Kenji Arisawa <arisawa@ar.aichi-u.ac.jp>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] fossil: disk is full
Date: Fri, 7 Mar 2003 17:43:55 +0900
Message-ID: <EE2274A4-5078-11D7-8202-000393A941BC@ar.aichi-u.ac.jp>

Sorry
 >I did that, but what happen if I have snapshot?
I did that, but what happen if I have no snapshot?

Kenji Arisawa

Re: [9fans] fossil: disk is full

[rog]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

> Yes, you can get away with snap -a occasionally
> during the write buffer, perhaps combined with
> stop mkfs |rc and restarting it.

i've just replaced my laptop disk (the old one died, but i managed to
back it up), and i thought this might be the right time to start using
fossil rather than kfs.

having read the "setting up fossil" page on the wiki, i thought i'd
give myself a fossil partition of 1.5GB and a venti partition of 19GB.

when copying in the (2.4GB) backup mkfs archive, i tried to use the
above mentioned technique, so i ran something like:

	while() {
		echo snap -a >> /srv/fscons
		stop mkext | rc
		sleep 1
		start mkext | rc
		sleep 300
	}

i did this in a hurry, and realise now that sleeping for only 1 second
was naive - maybe it should have slept for 300?

anyway, i left it overnight, and came back to many disk full messages,
and a broken fossil (traceback attached).

it seems a pity there's no way of doing this that doesn't rely on
guesswork: perhaps one might be able to put fossil into a mode where
if the disk fills up it would trigger an automatic snap -a and block
writers until it had all been written to venti?

in the meantime, what sort of numbers in the above script are likely
to keep fossil from filling up?

  cheers,
    rog.

[-- Attachment #2.1: Type: text/plain, Size: 312 bytes --]

The following attachment had content that we can't
prove to be harmless.  To avoid possible automatic
execution, we changed the content headers.
The original header was:

	Content-Disposition: attachment; filename=fossil.trace
	Content-Type: text/plain; charset="US-ASCII"
	Content-Transfer-Encoding: 7bit

[-- Attachment #2.2: fossil.trace.suspect --]
[-- Type: application/octet-stream, Size: 2811 bytes --]

% db 1521
386 binary
page fault
/sys/src/libc/9sys/abort.c:6 abort/               	CMPL	0,$0
$C
abort() /sys/src/libc/9sys/abort.c:6 called from _assert+3a /sys/src/libc/port/_assert.c:12
_assert(s=35e79) /sys/src/libc/port/_assert.c:7 called from fileDecRef+38 /sys/src/cmd/fossil/file.c:1127
fileDecRef(f=8e7478) /sys/src/cmd/fossil/file.c:1121 called from fsSnapshot+82 /sys/src/cmd/fossil/fs.c:571
       	fileDecRef.p/         	0x8eab98
       	fileDecRef.qq/         	0x0
fsSnapshot(fs=7d668, doarchive=1) /sys/src/cmd/fossil/fs.c:460 called from fsysSnap+74 /sys/src/cmd/fossil/9fsys.c:345
       	fsSnapshot.dst/         	0x8e7478
       	fsSnapshot.src/         	0x59968
fsysSnap(argv=7fffce40, argc=0, fsys=59a28) /sys/src/cmd/fossil/9fsys.c:329 called from fsysXXX+12e /sys/src/cmd/fossil/9fsys.c:1407
       	fsysSnap.usage/         	0x37739
       	fsysSnap.doarchive/         	0x1
       	fsysSnap._argc/         	0x6b680061
       	fsysSnap._args/         	0x56b4e
fsysXXX(argv=7fffce3c, name=8c9aa0, argc=2) /sys/src/cmd/fossil/9fsys.c:1376 called from cmdFsysXXX+3f /sys/src/cmd/fossil/9fsys.c:1423
       	fsysXXX.i/         	0xe
       	fsysXXX.fsys/         	0x59a28
       	fsysXXX.r/         	0x4f268
cmdFsysXXX(argv=7fffce3c, argc=2) /sys/src/cmd/fossil/9fsys.c:1414 called from cliExec+e0 /sys/src/cmd/fossil/Ccli.c:58
cliExec(buf=4ad20) /sys/src/cmd/fossil/Ccli.c:37 called from consProc+23c /sys/src/cmd/fossil/Ccons.c:304
       	cliExec.p/         	0x56b48
       	cliExec.argv/         	0x56b48
       	cliExec.argc/         	0x2
       	cliExec.i/         	0x11
       	cliExec.r/         	0x1
       	cliExec..safe/         	0x8e7678
consProc() /sys/src/cmd/fossil/Ccons.c:223 called from vtThread+30 /sys/src/libventi/plan9-thread.c:63
       	consProc.q/         	0x4afe8
       	consProc.n/         	0x8
       	consProc.buf/         	0x70616e73
       	consProc.r/         	0x14
       	consProc..safe/         	0x2000
       	consProc.i/         	0x7
       	consProc.wbuf/         	0x0
       	consProc.argv/         	0x20f72
       	consProc.argc/         	0x14
       	consProc.lp/         	0x0
vtThread(rock=0, f=1ac8) /sys/src/libventi/plan9-thread.c:47 called from consInit+44 /sys/src/cmd/fossil/Ccons.c:384
consInit() /sys/src/cmd/fossil/Ccons.c:376 called from main+117 /sys/src/cmd/fossil/fossil.c:72
main(argv=7fffefd4, argc=0) /sys/src/cmd/fossil/fossil.c:22 called from _main+31 /sys/src/libc/386/main9.s:16
       	main.cmd/         	0x4afc8
       	main.tflag/         	0x0
       	main.ncmd/         	0x1
       	main._argc/         	0x63
       	main._args/         	0x386f3
       	main.p/         	0x7fffeff2
       	main.i/         	0x0
       	main..safe/         	0x0--upas-pwzwxxhvwzolfvgkhmsqwfjljx--

Re: [9fans] fossil: disk is full

[Axel Belinfante]

I'm ``happy'' I did not try this before you did...
thanks for the implicit warning.

> it seems a pity there's no way of doing this that doesn't rely on
> guesswork: perhaps one might be able to put fossil into a mode where
> if the disk fills up it would trigger an automatic snap -a and block
> writers until it had all been written to venti?

My thought exactely.

Axel.

P.S. our spamassasin classified your message as spam, one major
reason seems for that to be a missing newline at the end of the trace
before (what should be a line) --upas-pwzwxxhvwzolfvgkhmsqwfjljx--
in my copy the last lines of your message are:

       	main.i/         	0x0
  	main..safe/         	0x0--upas-pwzwxxhvwzolfvgkhmsqwfjljx--

Re: [9fans] fossil: disk is full

[Russ Cox]

> P.S. our spamassasin classified your message as spam, one major
> reason seems for that to be a missing newline at the end of the trace
> before (what should be a line) --upas-pwzwxxhvwzolfvgkhmsqwfjljx--
> in my copy the last lines of your message are:
> 
>        	main.i/         	0x0
>   	main..safe/         	0x0--upas-pwzwxxhvwzolfvgkhmsqwfjljx--

Dave fixed this in upas a few weeks ago but it didn't
make it out to sources.  I just pushed the files out.
New binaries will appear this evening.

Russ

Re: [9fans] fossil: disk is full

[Russ Cox]

I can't figure out how you got that stack trace,
even if the disk is full.

Re: [9fans] fossil: disk is full

[Russ Cox]

If you've got 2.4GB of data and 1.5GB of write buffer,
you should be able to get away with one or two snap -a's
in the course of the copy.  There should probably be a
flush-on-demand, but there isn't yet.

Russ

Re: [9fans] fossil: disk is full

[Axel Belinfante]

For the record (or the archives), I'd like to mention that for me
the numbers in the following (conservative?) script allow me to
copy > 20 Gb to fossil with a disk cache of 1 Gb and some 50Gb
of venti. The number are conservative in the sense that I tried
to stop mkext long enough for the snap -a to finish.
It seems I did not succeed in that, but at least it seems that
one 'snap -a' finishes before the next one is started.
In some 24 hours something like 12 Gb gets copied.

Different numbers in the script and/or different cache size
command line flags for fossil-open -c and for venti (see below)
might make it faster (doing more in parallel); for me right now
(after a runaway script that flooded fscns with 'snap -a' requests
and thus broke fossil) the main thing is that it gets the (copying)
job done.

 while () {
	date
    echo sleep 1800
    sleep 1800
	date
    echo stop mkext
    stop mkext |rc
	echo snap -a
	echo snap -a >> /srv/fscons
	date
    echo sleep 1800
    sleep 1800
	date
    echo start mkext
    start mkext | rc
}

I already had set the command line option cache numbers
before I asked about the ratios here. I had made a similar
calculation as Russ showed, but taken less space for the
user and forgotten about the default kernelpercent.
Currently I'm using fossil-open -c 12800; venti (-I -B -C)^' '^128m
which probably explains why I see in stats that mem is fully
used, and also swap space is used.
At the start of each copy-to-venti swap usage increases almost to
the max and slowly decreases until the block is copied after which
it increases again, etc.  stats really nicely shows the copying of
the blocks to venti, and the copying with mkext to fossil.

Oops. while typing this I see an error passing by; I mention
it in a separate message.


Axel.

> in the meantime, what sort of numbers in the above script are likely
> to keep fossil from filling up?