Re: [9fans] Fs64 file server, partition boundaries out of range

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

Here's a dumb question. I have been copying info from my file server
console. Geoff has mentioned the file server log a couple of times. I
assumed it would be in /sys/log, but I can't find it. In order to get
all of the info from check, I either need a log to cut and paste from
or a paging command to move back on the console.

Greg

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

You have to set up console logging yourself on a cpu server, like this:

	aux/clog /mnt/consoles/fileserver /sys/log/fileserver &

where "fileserver" is the name of your file server.  This assumes
you've already got consolefs(4) configured, running and mounted, and
that you've got the file server's serial console configured and wired
up to a serial port on the cpu server doing the logging.  Add this to
your file server's plan9.ini:

console=0 baud=9600

It's a bit of work to set up, but very handy.  Not only do you get a
console log, but you can access the console from any of your Plan 9
machines with:

	C fileserver

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

Console setup definitely seems worthwhile, but it will take me a
little time to get to it.

On the off chance I got some useful output from check, here it is. I
don't think too much scrolled by.

check: "/386/bin/upas/filter": xtag: checktag
	tag=Tnone/0; expected Tind1/50352 -- flushed (225420)
check: "/386/bin/upas/list": xtag: checktag
	tag=Tnone/0; expected Tind1/50391 -- flushed (225953)
check: "/386/bin/upas/_marshal": xtag: checktag
	tag=Tnone/0; expected Tind1/50406 -- flushed (226221)
check free list
lo=656348	hi=703242
nfiles=43599
fsize=712733
nused=201282
ndup=0
nfree=38803
tfree=38803
nfdup=0
nmiss=472646
nbad=0
nqbad=0
maxq=167269
base stack=620
high stack=1920 of 16000
deepest recursion=13

That's from the following config. When I did copyworm, I just had h0
as main and w2 as output; otherwise I get the no blocks to copy
message.

config w0
service pinky
filsys main c[w0w1]f{h0h2}
filsys dump o
ipauth 192.168.0.109
ip 192.168.0.108
ipgw 192.168.0.1
ipmask 255.255.255.0

Just for the record, I tried copyworm from h0 to (w2w3) giving me
approximately 100G target. There were no error messages at the end of
copyworm, but when I tried to boot c[w0w1]f(w2w3) I got

tag=Tnone/0; expected Tvirgo/12211541 - flushed (12211541)
panic:fworm:checktag 12211541
cpu 0 exiting

which is like what I started with only apparently reflecting the
larger size

Greg

[-- Attachment #2: Type: message/rfc822, Size: 2895 bytes --]

From: geoff@plan9.bell-labs.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] Fs64 file server, partition boundaries out of range
Date: Mon, 7 Aug 2006 17:29:16 -0400
Message-ID: <ad3def584bb37a8e94c50bf880db6ce8@plan9.bell-labs.com>

You have to set up console logging yourself on a cpu server, like this:

	aux/clog /mnt/consoles/fileserver /sys/log/fileserver &

where "fileserver" is the name of your file server.  This assumes
you've already got consolefs(4) configured, running and mounted, and
that you've got the file server's serial console configured and wired
up to a serial port on the cpu server doing the logging.  Add this to
your file server's plan9.ini:

console=0 baud=9600

It's a bit of work to set up, but very handy.  Not only do you get a
console log, but you can access the console from any of your Plan 9
machines with:

	C fileserver

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

I think I understand your problem now.  If you were copying a fake
worm to a device of identical size, you could use copydev from h0, but
when copying a fake worm, you need to use copyworm from fh0 or
possibly even your normal main (c[w0w1]f{h0h2}).

check reports an fsize of 712733, so that's the highest block number
so far consumed by your fake worm, and is what copyworm should print
as your limit (plus or minus a few blocks) when you copy from fh0 or
your normal main.  When you copied from h0, copyworm just used the
size of h0 as its upper limit.

Perhaps copyworm should complain if the source file system does not
contain a worm.

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

Just to follow up. When I try to copyworm with c[w0w1]f{h0h2}, or
f{h0h2}, or fh0 I get the following right after `beginning copyworm in
8 seconds'

fworm: read 0
i/o error reading fh0 block 0
panic: no blocks to copy on fh0
cpu 0 exiting

I should say that things like history, yesterday, 9fs dump have
been working as advertised, so I do have a functioning worm.

I also tried to make main h0 and be explicit about the fake worm-ness
of output, which I hadn't been before (filsys output f{w2w3}). This
was definitely different. For one, it took even longer than my other
tries. But, instead of the `copyworm finished ... looping ... reset
any time' message, it had rebooted itself.

I'm going to stop this effort for a little while. At least until I get
console logging set up, so that I can be sure to be accurate when
I report what happened.

Thanks again.

Greg

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Geoff Collyer 7 Aug 2006 wrote:

I think I understand your problem now.  If you were copying a fake
worm to a device of identical size, you could use copydev from h0, but
when copying a fake worm, you need to use copyworm from fh0 or
possibly even your normal main (c[w0w1]f{h0h2}).

check reports an fsize of 712733, so that's the highest block number
so far consumed by your fake worm, and is what copyworm should print
as your limit (plus or minus a few blocks) when you copy from fh0 or
your normal main.  When you copied from h0, copyworm just used the
size of h0 as its upper limit.

Perhaps copyworm should complain if the source file system does not
contain a worm.

[-- Attachment #2: Type: message/rfc822, Size: 2895 bytes --]

From: geoff@plan9.bell-labs.com
To: 9fans@cse.psu.edu
Subject: Re: [9fans] Fs64 file server, partition boundaries out of range
Date: Mon, 7 Aug 2006 17:29:16 -0400
Message-ID: <ad3def584bb37a8e94c50bf880db6ce8@plan9.bell-labs.com>

You have to set up console logging yourself on a cpu server, like this:

	aux/clog /mnt/consoles/fileserver /sys/log/fileserver &

where "fileserver" is the name of your file server.  This assumes
you've already got consolefs(4) configured, running and mounted, and
that you've got the file server's serial console configured and wired
up to a serial port on the cpu server doing the logging.  Add this to
your file server's plan9.ini:

console=0 baud=9600

It's a bit of work to set up, but very handy.  Not only do you get a
console log, but you can access the console from any of your Plan 9
machines with:

	C fileserver

Re: Re: [9fans] Fs64 file server, partition boundaries out of range

[Anthony Sorace]

defining "ordinary" is, of course, impossible, but about yea fast:
http://www.cs.bell-labs.com/who/seanq/p9trace.html

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

Well, don't I feel silly. I thought I had stumbled upon a problem
that required an fs wizard, but, unless Geoff says otherwise, I'm
now convinced that I was just trying to get an elephant into a
hamster cage. I'll just have to rethink my disk usage. Or maybe I'll
try fossil/venti. Fs has been great, but the graph indicates that
venti grows more slowly, and I don't want to give up my mirror for
concatenation.

Now, the question is, if I switch to venti, how do I do it without
losing my history? Seems to me that I've seen posts saying something
like "Nemo posted a script", but I've never found the script. But
that's just a vague memory. I'll have to do some 9fans archive
searching.

Thanks.

Greg

Re: [9fans] Fs64 file server, partition boundaries out of range

[erik quanstrom]

i'd really be suprised if you had actually used up 50G.  i think that
the problem is that copyworm trys to copy the whole worm --- even
empty blocks.

the relevant check in getbuf combined with written size seems to do
this:

	static Devsize
	writtensize()
	{
		Devsize lim;

		for(lim = devsize(worm); lim > 0; lim--)
			if(blocknum(lim) is "active" OR blocknum(lim) can be read)
				return lim+1;
		return 0;
	}

am i missing something?  it's not obvious to me how to determine the last
block that's actually got data in it.

- erik

On Sat Aug  5 15:22:54 CDT 2006, g.pavelcak@comcast.net wrote:
> 
> Well, don't I feel silly. I thought I had stumbled upon a problem
> that required an fs wizard, but, unless Geoff says otherwise, I'm
> now convinced that I was just trying to get an elephant into a
> hamster cage. I'll just have to rethink my disk usage. Or maybe I'll
> try fossil/venti. Fs has been great, but the graph indicates that
> venti grows more slowly, and I don't want to give up my mirror for
> concatenation.
> 
> Now, the question is, if I switch to venti, how do I do it without
> losing my history? Seems to me that I've seen posts saying something
> like "Nemo posted a script", but I've never found the script. But
> that's just a vague memory. I'll have to do some 9fans archive
> searching.
> 
> Thanks.
> 
> Greg

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

I believe Charles wrote the original version of what is now
dowormcopy() and I probably modified it; Nigel may have also.  It
copes with real and fake worms.  On a fake worm, the block-allocation
bitmap is at the end of the file system, but all blocks can be read.

On a real worm, only already-written blocks can be read and there is
no allocation bitmap (or at least none visible outside the jukebox
hardware).  Because the file server kernel gradually extends the upper
limit of blocks it will allocate on the worm, there will tend to be
both written and unwritten blocks just before the current nominal end
of the file system (but the end is increased when cwgrow() is called).
After the current end there should only be unwritten blocks, and
almost all of the blocks before it should be written.

All sizes are in blocks of RBUFSIZE bytes.  writtensize() gets the
device's size from devsize(), which calls fwormsize() for fake worms
or cwsize() for real ones.  cwsize() just gets the current end of the
file system from the cache device, where it's conveniently stored.
fwormsize() subtracts the size of the allocation bitmap from the size
of the underlying device.  writtensize() works backward from the end
of the device, using the size returned by devsize() as its nominal
size, reading blocks until it succeeds.  This should be pretty quick;
for fake worms, fwormread() just consults the allocation bitmap to see
if a block is allocated and thus readable, so writtensize() will see a
long series of failed getbuf() attempts before reading the last
allocated block before the allocation bitmap.  Real worms are similar,
but starting at the cache's notion of where the file system currently
ends will be much closer to the last-written block than starting at
the end of the non-bitmap fake worm device, so the backward scan will
be short.

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

I didn't explain that as clearly as I could have: all blocks on the
disk underlying a fake worm can be read from the disk device, but only
already-written blocks can be read via the fake-worm device, and it's
the fake worm's allocation bitmap that determines which blocks have
been written.  This in turn emulates the real worm hardware, which
only permits reading written blocks.

Re: [9fans] Fs64 file server, partition boundaries out of range

[erik quanstrom]

you're explination made perfect sense to me.  where is the bitmap consulted?

- erik

On Sat Aug  5 18:22:14 CDT 2006, geoff@plan9.bell-labs.com wrote:
> I didn't explain that as clearly as I could have: all blocks on the
> disk underlying a fake worm can be read from the disk device, but only
> already-written blocks can be read via the fake-worm device, and it's
> the fake worm's allocation bitmap that determines which blocks have
> been written.  This in turn emulates the real worm hardware, which
> only permits reading written blocks.
> 

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

Good; when I reread my first message, it looked imprecise enough to
confuse.

/sys/src/fs/dev/fworm.c contains the fake worm implementation and
fwormread() and fwormwrite() consult and maintain the bitmap of
written blocks.

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

I hate to be the slowest guy in the room, but it's not clear where
this leaves me. If I understand the paragraph below, the upshot is
that there may be some unwritten blocks that copyworm thinks it has
to copy, but it's not the case that it is just trying to copy all
80G worth, written or not, onto my 50G disk. So, I either really do
have over 50G of blocks that it's trying to copy, or there is some
other problem. I see that the limit() value Geoff asked for is
basically devsize() for my source disk. Does that mean it's just too
much stuff?

Thanks for all the help.

Greg

> 
> On a real worm, only already-written blocks can be read and there is
> no allocation bitmap (or at least none visible outside the jukebox
> hardware).  Because the file server kernel gradually extends the upper
> limit of blocks it will allocate on the worm, there will tend to be
> both written and unwritten blocks just before the current nominal end
> of the file system (but the end is increased when cwgrow() is called).
> After the current end there should only be unwritten blocks, and
> almost all of the blocks before it should be written.
> 

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

copyworm shouldn't copy unwritten blocks.  I'm still not sure that I
know what the problem here is.  Could you run these commands on your
file server console and send us the output?

	cfs main
	check
	printconf

The file server will be unresponsive until check finishes, so don't be
alarmed.

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

[-- Attachment #1: Type: text/plain, Size: 2324 bytes --]


I finally got to check on this.

First thanks to Geoff for his informative replies and to jmk for  
taking the
time to let me know Geoff was away. That saved me from checking my
email every 5 seconds looking for his posts.

I tried copyworm yesterday. I wasn't sure what device I was supposed to
use. First I made main fh0 and output fw2, but I got a message saying
something like `no blocks to copy from worm'. So, then I just made main
h0 and output w2. Copyworm ran and ended with (and I actually wrote the
message down this time):

wreniocmd out of range a=w2 b=6105859
wrenwrite: w2(6105859) bad status 0040
out block 6105859: write error; bailing copied 615859 blocks from h0  
to w2
sync: wormcopy
looping: reset the machine at any time

h0 is an 80G IDE drive (Western Digital something), and w2 is a Seagate
50GB U2WCS 3HH ST150176LC 1.6-IN HIGH 80-PIN
according to the seller's blurb.

I  wouldn't think 50 is just too small given that I only had a few GB  
of stuff and
have only been running since December.

I need to try the p(w2)1.99. I didn't have any intention of having an  
MBR
but maybe something there is the problem. Do you think that might help?
It takes almost 20hrs to complete the copy, so it's not an easy  
experiment
to do.

Anyway, any thoughts appreciated.

Greg




-=-=-=-=-=-=-=-=-=-=-=-=-
It sounds like

        cannot add sd12!plan9 [63,156296385] to disk [0,976937355):
partition boundaries out of range

is coming from 9load.  I haven't seen this, but it does sound like
garbled partition tables or a bad disk.  The first block of a file
system contains its configuration block, so if you plan to have an MBR
or other DOS/FAT stuff on the disk, you'll need to skip it.  I do that
on my main file server to allow a small DOS partition.  To do so, use
p(w0)1.99 instead of w0, for example.

`panic: fworm: checktag 6105775' is quite serious.  The fake-worm
bitmap isn't initialised.  Copying fworms is tricky, particularly if
the output device is larger than the input device.  You probably ought
to use copyworm instead of copydev, since copydev just blindly copies
blocks, assuming input and output are the same size, but copyworm
knows how to copy fworms, including reaming the output fworm, thus
creating its bitmap.


[-- Attachment #2: Type: text/html, Size: 9184 bytes --]

Re: [9fans] Fs64 file server, partition boundaries out of range

[erik quanstrom]

i don't have a worm or pseudoworm, but it's interesting that 
50G/8k blocksize is approximately 6105859 (i get 6103515+5/8).
is your total size of your original worm >50G?  i'm pretty sure
copyworm just does a block-for-block copy and doesn't check to see
if blocks are used or not.  also, the error code matches this code from
wreniocmd:

/sys/src/fs/dev/wren.c:83,86; #1566,#1668
	if(b >= dr->max) {
		print("wreniocmd out of range a=%Z b=%lld\n", d, (Wideoff)b);
		return 0x40;
	}

so i think it's pretty likely that your pseudoworm is too large for the new disk.

- erik

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

That would be funny if that's all it was. And surprising to me. How  
fast does a worm grow under "ordinary" usage?

Greg

On Aug 5, 2006, at 9:57 AM, erik quanstrom wrote:

> i don't have a worm or pseudoworm, but it's interesting that
> 50G/8k blocksize is approximately 6105859 (i get 6103515+5/8).
> is your total size of your original worm >50G?  i'm pretty sure
> copyworm just does a block-for-block copy and doesn't check to see
> if blocks are used or not.  also, the error code matches this code  
> from
> wreniocmd:
>
> /sys/src/fs/dev/wren.c:83,86; #1566,#1668
> 	if(b >= dr->max) {
> 		print("wreniocmd out of range a=%Z b=%lld\n", d, (Wideoff)b);
> 		return 0x40;
> 	}
>
> so i think it's pretty likely that your pseudoworm is too large for  
> the new disk.
>
> - erik

Re: [9fans] Fs64 file server, partition boundaries out of range

[geoff]

At the beginning of the copyworm operation, writtensize() should do this:

	print("limit(%Z) = %lld\n", worm, (Wideoff)lim);

It would help to know what that line was when you ran copyworm.

Re: [9fans] Fs64 file server, partition boundaries out of range

[Gregory Pavelcak]

limit(h0) = 9768447

I copied all of the devinit  stuff if you need anything else, but I  
won't bother typing it unless you want it.

Thanks.

Greg

On Aug 5, 2006, at 10:38 AM, geoff@plan9.bell-labs.com wrote:

> At the beginning of the copyworm operation, writtensize() should do  
> this:
>
> 	print("limit(%Z) = %lld\n", worm, (Wideoff)lim);
>
> It would help to know what that line was when you ran copyworm.
>