[9fans] venti wrarena i/o errors

[9fans] venti wrarena i/o errors

[Robert Raschke]

Hi,

I am trying to migrate my old venti/fossil from one machine to another
(the new machine is running the new venti).  I am doing this pretty
piecemeal and by hand to find my bearings.  When I'm running
venti/wrarena on the old machine pointing at the new one, I am getting
i/o errors on the new machine.  I'm wondering if these point to disk
problems, or if they are benign?

On "new" I created the partitions and did venti/fmtarenas,
venti/fmtisect, venti/fmtindex and finally venti/venti with the
following venti.conf:
index main
isect /dev/sdC0/isect
arenas /dev/sdC0/arenas
mem 5M
bcmem 10M
icmem 15M

On "old", I saved the first arena into a file using venti/rdarena.
And then I'm running
venti/wrarena -h new arenas0
on "old" and the copying starts.

But on "new" I get these errors like these during the copy:
2007/1128 22:10:54 err 4: write /dev/sdC0/arenas offset 0x312a000 count 8192 buf 3458000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x312a000: write error: write /dev/sdC0/arenas offset 0x312a000 count 8192 buf 3458000 returned -1: i/o error
2007/1128 22:10:55 err 4: read /dev/sdC0/isect offset 0x7767e000 count 8192 buf 3836000 returned -1: i/o error
2007/1128 22:18:50 err 4: write /dev/sdC0/arenas offset 0x6c86000 count 8192 buf 3c0c000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x6c86000: write error: write /dev/sdC0/arenas offset 0x6c86000 count 8192 buf 3c0c000 returned -1: i/o error
2007/1128 22:18:50 err 4: read /dev/sdC0/isect offset 0x5d686000 count 8192 buf 348c000 returned -1: i/o error
2007/1128 22:18:50 err 4: write /dev/sdC0/arenas offset 0x6c88000 count 8192 buf 353e000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x6c88000: write error: write /dev/sdC0/arenas offset 0x6c88000 count 8192 buf 353e000 returned -1: i/o error
2007/1128 22:22:50 err 4: write /dev/sdC0/arenas offset 0x9e9e000 count 8192 buf 33ac000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x9e9e000: write error: write /dev/sdC0/arenas offset 0x9e9e000 count 8192 buf 33ac000 returned -1: i/o error
2007/1128 22:22:51 err 4: read /dev/sdC0/isect offset 0x139dc000 count 8192 buf 39b0000 returned -1: i/o error
2007/1128 22:22:51 err 4: write /dev/sdC0/arenas offset 0x9ea0000 count 8192 buf 3c00000 returned -1: i/o error
2007/1128 22:22:51 err 4: read /dev/sdC0/isect offset 0x36c6c000 count 8192 buf 3b1c000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x9ea0000: write error: write /dev/sdC0/arenas offset 0x9ea0000 count 8192 buf 3c00000 returned -1: i/o error
2007/1128 22:34:58 err 4: write /dev/sdC0/arenas offset 0x19a06000 count 8192 buf 3312000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x19a06000: write error: write /dev/sdC0/arenas offset 0x19a06000 count 8192 buf 3312000 returned -1: i/o error
2007/1128 22:34:58 err 4: read /dev/sdC0/isect offset 0x85130000 count 8192 buf 3626000 returned -1: i/o error
2007/1128 22:34:58 err 4: write /dev/sdC0/arenas offset 0x19a08000 count 8192 buf 349e000 returned -1: i/o error
venti/venti: writeproc: part /dev/sdC0/arenas addr 0x19a08000: write error: write /dev/sdC0/arenas offset 0x19a08000 count 8192 buf 349e000 returned -1: i/o error


So, I am wondering if I've got a duff disk, or if there is something
else that I can investigate.  Any pointers are gratefully received.

Robby

--
r.raschke can be found at tombob.com .

Re: [9fans] venti wrarena i/o errors

[erik quanstrom]

forgive the silly question, but have you verified that these offsets are
not beyond the end of the partition?

you can test reading partitions easily with

	dd -if /dev/sdC0/isect -of /dev/null -bs 64k

it wouldn't be too hard to write a scrubber that reads and
then rewrites the disk.  it should try the rewrite twice
in case the bad block allocation policy is to return
a write error and then allocate a spare block.
(this is the case with many drives.)

- erik

Re: [9fans] venti wrarena i/o errors

[ron minnich]

On Dec 14, 2007 12:22 AM, Christian Kellermann
<Christian.Kellermann@nefkom.net> wrote:

> This looks really good! Do you still have to do the
> replace-the-chip-while-power-on dance to get the LinuxBIOS flashed?
> They don't come with it natively as it seems...

it's soldered on :-)

so you flash it and say your prayers.

ron

Re: [9fans] venti wrarena i/o errors

[Richard Miller]

> Now, if
> anyone knows of reliable, allowed to be slow, preferably cheap
> components I could use to set up a fanless Plan 9 box, I'd be
> delighted to hear from you.

I've used the Via MS10000E with no fan and no problems.

Re: [9fans] venti wrarena i/o errors

[Christian Kellermann]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

* ron minnich <rminnich@gmail.com> [071214 01:50]:
> I can highly recommend these:
> http://www.pcengines.ch/alix1c.htm
> 
> The company is great to deal with. They run linuxbios now and I'm
> looking at embedding plan 9.
> 
> Price is $134.

This looks really good! Do you still have to do the
replace-the-chip-while-power-on dance to get the LinuxBIOS flashed?
They don't come with it natively as it seems...

Regards,

Christian

-- 
You may use my gpg key for replies:
pub  1024D/47F79788 2005/02/02 Christian Kellermann (C-Keen)

[-- Attachment #2: Type: application/pgp-signature, Size: 194 bytes --]

Re: [9fans] venti wrarena i/o errors

[ron minnich]

I can highly recommend these:
http://www.pcengines.ch/alix1c.htm

The company is great to deal with. They run linuxbios now and I'm
looking at embedding plan 9.

Price is $134.

ron

Re: [9fans] venti wrarena i/o errors

[sqweek]

On Dec 11, 2007 6:41 PM, Robert Raschke <rrplan9@tombob.com> wrote:
> Looks like I'll have to live with fan noise a little longer.  Now, if
> anyone knows of reliable, allowed to be slow, preferably cheap
> components I could use to set up a fanless Plan 9 box, I'd be
> delighted to hear from you.

 I ran into http://www.fit-pc.com/ recently, which is a lil' fanless
box. Can't vouch for its reliabilty or plan 9 compatibility or
anything as I don't own one... Motherboard spec is here:
http://www.compulab.co.il/iglx/html/iglx-sb-datasheet.htm
-sqweek

Re: [9fans] venti wrarena i/o errors

[Robert Raschke]

Hi,

just wanted to wind down this thread with my latest findings.

It turns out that some of the VIA EPIA boards have issues with DMA in
general.  If you dig in the viarena forums, you'll come across a
multitude of problems around streaming, DVD access, hard disk access,
etc.  I have now had difficulties not just with putting Plan 9 on
them, but also Free BSD.

Since I only paid ~$15 for the board, and I don't have the time and
energy to pursue the issues, I'm binning the board and will
investigate alternatives for a fanless setup at my leisure.

Looks like I'll have to live with fan noise a little longer.  Now, if
anyone knows of reliable, allowed to be slow, preferably cheap
components I could use to set up a fanless Plan 9 box, I'd be
delighted to hear from you.

Robby

--
It's r.raschke instead of rrplan9 for mail.

Re: [9fans] venti wrarena i/o errors

[Robert Raschke]

I think my disk issues might be heat related.  I tested the old venti
and it failed with an i/o error as well.  After I start getting
failures, the dd commands fail as well, in fact any disk access fails.
The common factor is that failures appear roughly after 30 minutes.

I will provide better airflow to the disk and see what happens then.

Robby

Re: [9fans] venti wrarena i/o errors

[Russ Cox]

> lock 0xf0c77cf8 loop key 0xdeaddead pc 0xf01c846f held by pc 0xf01c846f proc 307
> 295:     venti pc f01da773 dbgpc    203db     Pread (Running) ut 1 st 537 bss 4342000 qpc f01be14f nl 0 nd 0 lpc f01c57a1 pri 3
> 307:     venti pc f01cded7 dbgpc    203db     Pread (Ready) ut 332 st 1137 bss 4342000 qpc f013ea9a nl 2 nd 0 lpc f01c1026 pri 0
> lock 0xf0c77cf8 loop key 0xdeaddead pc 0xf01c846f held by pc 0xf01c846f proc 307
> 297:     venti pc f01cda6c dbgpc    203db     Pread (Running) ut 79 st 553 bss 4342000 qpc f01c8d59 nl 0 nd 0 lpc f01c57a1 pri 3
> 307:     venti pc f01cded7 dbgpc    203db     Pread (Ready) ut 332 st 1137 bss 4342000 qpc f013ea9a nl 2 nd 0 lpc f01c108e pri 0

Can you run:

	% acid /path/to/your/kernel
	acid: src(0xf01c846f)

and let us know what that prints?

> 2007/1204 22:24:20 err 4: write /dev/sdC0/isect offset 0x293ae000 count 65536 buf 337e000 returned -1: i/o error
> venti/venti: part /dev/sdC0/isect addr 0x2922e000: icachewritesect writepart: write /dev/sdC0/isect offset 0x293ae000 count 65536 buf 337e000 returned -1: i/o error
> 2007/1204 22:24:21 err 4: read /dev/sdC0/isect offset 0x29a2e000 count 65536 buf 31fe000 returned -1: i/o error

It looks very much like your disk has bad sectors or something like that.
Try running:

	dd -bs 65536 < /dev/sdC0/isect >/dev/null
	dd -bs 65536 < /dev/zero >/dev/sdC0/isect
	dd -bs 65536 < /dev/sdC0/isect >/dev/null

If the first dd fails, that would at least exonerate venti.
Either way, the second dd might get rid of any bad sectors
by overwriting them and letting the disk remap to some of
its reserve sectors (but it's probably time to replace the disk anyway).

You could also try running the dd immediately after venti fails,
in case it's something like the disk getting too hot.

I'd feel more confident your disk was bad if I understood the
lock loop above.  If the lock loop is something "impossible" then
it could be that the disk controller is just screwing with memory.

It's also possible that your disk cables can't handle the dma speeds
that Plan 9 is trying or that they are otherwise just not good enough.
The SP1613N looks like a laptop-sized disk, though, so maybe 
there isn't even a cable!

Russ

Re: [9fans] venti wrarena i/o errors

[Robert Raschke]

Hi,

I'm still having issues with Venti on a VIA EPIA-5000.  I am getting
write i/o errors from icachewritesect, as well as lock errors.

My disk is a Samsung SP1613N, and is fine according to Samsung's
HUTIL.  I also ran memtest86 in case my memory is sick, but that is
fine too.

I booted using PXE and my disk was completely clean.  My startup info
looks like this:

Plan 9
E820: 00000000 0009fc00 memory	E820: 0009fc00 000a0000 reserved
E820: 000f0000 00100000 reserved	E820: 00100000 0f7f0000 memory
E820: 0f7f0000 0f7f3000 acpi nvs	E820: 0f7f3000 0f800000 acpi reclaim
E820: ffff0000 100000000 reserved	126 holes free
00054000 0008a000 221184
002ee000 064cf000 102633472
102854656 bytes free
cpu0: 533MHz CentaurHauls Via C3 Samuel 2 or Ezra (cpuid: AX 0x0673 DX 0x803035)
ELCR: 0E08
pcirouting: ignoring south bridge PCI.0.0.0 1106/0601
#l0: vt6102: 100Mbps port 0xE800 irq 11: 004063e24ea8
#U/usb0: uhci: port 0xD400 irq 3
#U/usb1: uhci: port 0xD800 irq 3
248M memory: 101M kernel data, 147M user, 561M swap
root is from (tcp)[tcp]: 
user[none]: rtr
secstore password: 
version...time...

init: starting /bin/rc

Next I set up ~30GB on the disk:

term% disk/mbr -m /386/mbr /dev/sdC0/data
term% disk/fdisk /dev/sdC0/data
[...]
>>> p
'  p1                      0 3911         (3911 cylinders, 29.95 GB) PLAN9
   empty                3911 19457        (15546 cylinders, 119.08 GB) 
>>> w
>>> q
term% disk/prep /dev/sdC0/data
>>> p
' 9fat                   0 204800      (204800 sectors, 100.00 MB)
' nvram             204800 204801      (1 sectors, 512 B )
' fossil            204801 10192089    (9987288 sectors, 4.76 GB)
' bloom           10192089 10224857    (32768 sectors, 16.00 MB)
' arenas          10224857 60325137    (50100280 sectors, 23.88 GB)
' isect           60325137 62830152    (2505015 sectors, 1.19 GB)
>>> w
>>> q
term% disk/format -b /386/pbs -d -r 2 /dev/sdC0/9fat /386/9load /386/9pcf
add 9load at clust 2
add 9pcf at clust a7
Initializing FAT file system
type hard, 12 tracks, 255 heads, 63 sectors/track, 512 bytes/sec
Adding file /386/9load, length 337588
add 9load at clust 2
Adding file /386/9pcf, length 2874011
add 9pcf at clust a7
used 3215360 bytes

And now I set up venti:

term% cat /tmp/venti.conf
index main
isect /dev/sdC0/isect
arenas /dev/sdC0/arenas
bloom /dev/sdC0/bloom
mem 2M
bcmem 4M
icmem 6M
addr tcp!*!17034
httpaddr tcp!*!8000
term% venti/fmtisect isect /dev/sdC0/isect
fmtisect /dev/sdC0/isect: 156,466 buckets of 215 entries, 524,288 bytes for index map
term% venti/fmtarenas arenas /dev/sdC0/arenas
fmtarenas /dev/sdC0/arenas: 48 arenas, 25,650,544,640 bytes storage, 524,288 bytes for index map
term% venti/fmtbloom /dev/sdC0/bloom
fmtbloom: using 16MB, 32 hashes/score, best up to 2,982,616 blocks
term% venti/conf -w /dev/sdC0/arenas </tmp/venti.conf
term% venti/fmtindex /dev/sdC0/arenas
fmtindex: 48 arenas, 156,466 index buckets, 25,649,758,208 bytes storage
term% venti/venti -c /dev/sdC0/arenas
2007/1204 21:43:15 venti: conf...httpd tcp!*!8000...init...icache 6,291,456 bytes = 98,304 entries; 4 scache
sync...announce tcp!*!17034...serving.

So far so good, I previously saved my arenas on my old
auth/cpu/fossil/venti server (running the old venti in a cpuf kernel)
using the new venti/rdarena.  I now use the new venti/wrarena on my
old server and load the first arena into my new running venti.

The first error turns up on my screen a little while later:

lock 0xf0c77cf8 loop key 0xdeaddead pc 0xf01c846f held by pc 0xf01c846f proc 307
295:     venti pc f01da773 dbgpc    203db     Pread (Running) ut 1 st 537 bss 4342000 qpc f01be14f nl 0 nd 0 lpc f01c57a1 pri 3
307:     venti pc f01cded7 dbgpc    203db     Pread (Ready) ut 332 st 1137 bss 4342000 qpc f013ea9a nl 2 nd 0 lpc f01c1026 pri 0
lock 0xf0c77cf8 loop key 0xdeaddead pc 0xf01c846f held by pc 0xf01c846f proc 307
297:     venti pc f01cda6c dbgpc    203db     Pread (Running) ut 79 st 553 bss 4342000 qpc f01c8d59 nl 0 nd 0 lpc f01c57a1 pri 3
307:     venti pc f01cded7 dbgpc    203db     Pread (Ready) ut 332 st 1137 bss 4342000 qpc f013ea9a nl 2 nd 0 lpc f01c108e pri 0

Once the wrarena has finished and all activity has ceased on my new
machine I kill and restart venti (was reading the man pages in the
meantime, and thought turning on debug output might be a good idea)
(don't know if the sync does anything at all with the new venti):

term% venti/sync -h P9VIA
term% kill venti |rc
term% venti/venti -d -c /dev/sdC0/arenas
2007/1204 22:12:09 venti: conf...httpd tcp!*!8000...init...icache 6,291,456 bytes = 98,304 entries; 4 scache
sync...2007/1204 22:12:20 arenas0: indexing 96523 clumps...
announce tcp!*!17034...serving.

Now I get a sawtooth pattern in the load section of my stats window,
and after about 20 minutes of that, I get this in the window where I
started venti:

2007/1204 22:24:20 err 4: write /dev/sdC0/isect offset 0x293ae000 count 65536 buf 337e000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x2922e000: icachewritesect writepart: write /dev/sdC0/isect offset 0x293ae000 count 65536 buf 337e000 returned -1: i/o error
2007/1204 22:24:21 err 4: read /dev/sdC0/isect offset 0x29a2e000 count 65536 buf 31fe000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x29a2e000: icachewritesect readpart: read /dev/sdC0/isect offset 0x29a2e000 count 65536 buf 31fe000 returned -1: i/o error
2007/1204 22:24:21 err 4: read /dev/sdC0/isect offset 0x2a22e000 count 65536 buf 31fe000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x2a22e000: icachewritesect readpart: read /dev/sdC0/isect offset 0x2a22e000 count 65536 buf 31fe000 returned -1: i/o error
2007/1204 22:24:21 err 4: read /dev/sdC0/isect offset 0x2aa32000 count 65536 buf 31fe000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x2aa32000: icachewritesect readpart: read /dev/sdC0/isect offset 0x2aa32000 count 65536 buf 31fe000 returned -1: i/o error
2007/1204 22:24:28 err 4: read /dev/sdC0/isect offset 0x2b234000 count 65536 buf 31fe000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x2b234000: icachewritesect readpart: read /dev/sdC0/isect offset 0x2b234000 count 65536 buf 31fe000 returned -1: i/o error
2007/1204 22:24:29 err 4: read /dev/sdC0/isect offset 0x2ba36000 count 65536 buf 31fe000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x2ba36000: icachewritesect readpart: read /dev/sdC0/isect offset 0x2ba36000 count 65536 buf 31fe000 returned -1: i/o error
2007/1204 22:24:29 err 4: read /dev/sdC0/isect offset 0x2c236000 count 65536 buf 31fe000 returned -1: i/o error
venti/venti: part /dev/sdC0/isect addr 0x2c236000: icachewritesect readpart: read /dev/sdC0/isect offset 0x2c236000 count 65536 buf 31fe000 returned -1: i/o error

This looks like it will continue pretty much until I kill venti.

My drive info is:

term% cat /dev/sdC0/ctl
inquiry SAMSUNG SP1613N                         
config 0040 capabilities 2F00 dma 00550004 dmactl 00000000 rwm 16 rwmctl 0 lba48always off
geometry 312581808 512 16383 16 63
part data 0 312581808
part plan9 63 62830215
part 9fat 63 204863
part nvram 204863 204864
part fossil 204864 10192152
part bloom 10192152 10224920
part arenas 10224920 60325200
part isect 60325200 62830215

And attempting to turn on dma results in

atagenioretry: disabling dma
sdC0: retry: dma 00000000 rwm 0000

Which I guess means that I won't be using dma. That's ok.

The disk offsets into the isect partition look fine to me.

I guess I will try and cobble together the old venti from my archives
and give that a try.  Just to see if that behaves any different.  But
if anyone has any ideas what to try next, I'd love to hear
suggestions.

Thanks,
Robby


--
replace my plan9 mail alias with r.raschke for direct emails

Re: [9fans] venti wrarena i/o errors

[Joshua Wood]

> ...sources/contrib/steve/doc/Venti-rescue.pdf is a mildly-dated but  
> excellent...

I was only half right. The document is not even mildly dated. It is  
however indeed excellent.

If only I could work 'ls -l' I would have seen that Steve's version  
of the Venti-rescue.pdf file on sources/contrib is as recent as 11/27.

--
Josh

Re: [9fans] venti wrarena i/o errors

[Joshua Wood]

>
> The new machine is a VIA.  I will try again without dma.

Were you manually turning dma on? If not, this is not likely to be a  
case where your problem was same as the one I mentioned. I was  
forcing dma on a chipset not listed in the distributed /sys/src/9/pc/ 
sdata.c.

> I will also
> add in the bloom partition, but I'm low on memory at the moment (256MB
> doesn't sound low to me, but I find the new venti can be a bit greedy
> if not constrained), so I'm not sure if that is actually worth doing.

It's optional, and it's not causing your problem, so I don't want to  
distract from the immediate issue, but you can kinda get the best of  
both worlds if memory is tight by starting with a bloom filter, then  
dropping it from your venti conf as the system ages toward less write- 
intensiveness.

> I will read the venti man pages more slowly to understand better.

sources/contrib/steve/doc/Venti-rescue.pdf is a mildly-dated but  
excellent supplement to the manual for the task you're attempting,  
and discusses memory allotment between venti and fossil in an  
appendix, too.

--
Josh

Re: [9fans] venti wrarena i/o errors

[Robert Raschke]

Thank you for these ideas.

I will have to look at the offsets again tonight, just to make sure
that they look right.

The new machine is booted off the old venti, so the executables are
the same on both.

The new machine is a VIA.  I will try again without dma.  I will also
add in the bloom partition, but I'm low on memory at the moment (256MB
doesn't sound low to me, but I find the new venti can be a bit greedy
if not constrained), so I'm not sure if that is actually worth doing.
I will read the venti man pages more slowly to understand better.

I will also try the dd approach to do a simple test of the disk.

Thanks again,
Robby

Re: [9fans] venti wrarena i/o errors

[Joshua Wood]

If you suspect your hardware, you should verify it with an  
appropriate low-level tool, like a bootable utilities image from your  
drive manufacturer or similar (or memtest for memory, or etc, - using  
a low-level diagnostic on hardware can really speed up and focus  
software troubleshooting).

Given working hardware, according to the support in Plan 9 for your  
IDE or SATA controller, do you have DMA on when it shouldn't be? I  
saw i/o errors from venti when trying to force dma on a not-quite- 
supported via controller we have. If 'new' is a fresh install from a  
recent iso, you might want to ensure 'old' has been pull'd to the  
latest venti and rd/wrarena code, or invoke wrarena on 'new' itself.  
Lastly, does the venti on 'new' work for a basic operation, like vac(1)?

This month I have tested the procedure you're trying 4 or 5 times,  
and so if it is encouraging, the basic software and process works as  
expected/documented in venti-backup(8). I notice the lack of a bloom  
filter in your venti conf; I am not implying this is the direct  
source of your problem, but you should check out the latest venti(8)  
and its see also's. The bloom filter is probably most critical to  
performance, from what has been reported here and what I've seen, in  
a new initialization scenario, so you'll probably want one.

Since your