[9fans] New venti install won't boot after 05:00 crash

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

Hi,

Subsequent to some discussion on this list, I decided to try setting up
a vent-backed fossil file system.  I did this straight from the 4th
Ed. install CD.  The system booted from the HDD and seemed to start up
and shut down fine.  Then, one evening I used Acme to take some notes.
At about 05:00, I was bragging to a buddy of mine how cool Plan 9 was.
No sooner than I began to show him Acme's nifty chording, did the rio
window began filling up with error messages about trouble writing to the
hard drive.  Although rio kept running, the file system hung.  Not being
able to recover the system, I rebooted it.  Well, I tried to.  The boot
process didn't get very far.  Ever since then, every time I try to boot
it, the kernel panics.  (Messages below.)

If I knew anything about how the root filesystem was brought online
during boot, I might be able to debug this myself.  Alas, I have no idea
which programs are generating which parts of this output, so I really
have no angle from which to attack the problem.  Any ideas?  The P9
partition is the third primary partition on the disk.  I'm fairly sure
the HDD is good, because I tested it before the install, and have
installed and run Linux on the same HDD without incident.  Although I
started and shut down Plan 9 a number of times (just to test its range
of motion), this crash occurred the first time I tried saving any data
to the file system (a text file under /usr/glenda/).

Any help would be much appreciated!

Here's what the console shows (typed by hand, may be typos):

root is from (tcp, local)[local!#S/sdC0/fossil]: 
user[none]: glenda
time...
venti...2011/0318 00:24:56 venti: conf.../boot/venti: mem 6,252,817 bcmem 12,505,634 icmem 18,758,451...httpd tcp!127.1!8000...init...icache 18,758,451 bytes = 293,101 entries; 4 scache
sync...2011/0318 00:24:56 arenas00: indexing 45372 clumps...
announce tcp!127.1!17034...serving.
fossil(#S/sdC0/fossil)...fsys: dialing venti at tcp!127.1!17034
/boot/fossil: labelUnpack: bad label: 0xcb 0xb2 0x3d22d532 0x8cacb517 0xfc0d00cb
/boot/fossil: fsOpen error
fsOpen: corrupted block label
fsys main open  -c 3000: fsOpen: corrupted block label
fsys: 'main' not open
fsys main snaptime -s 60 -a 0500 -t 2880: fsys 'main' not open
version...boot: mount /: fsys: 'main' not open
panic: boot process died: unknown
panic: boot process died: unknown
dumpstack disabled
cpu0: exiting

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[John Floren]

After the initial install, did you reboot pretty soon? Venti wants to spend
some time writing an initial image after the first boot, it seems, and if
you interrupt that you're in trouble

On Mar 17, 2011 6:00 PM, <smiley at zenzebra.mv.com> wrote:
> Hi,
>
> Subsequent to some discussion on this list, I decided to try setting up
> a vent-backed fossil file system. I did this straight from the 4th
> Ed. install CD. The system booted from the HDD and seemed to start up
> and shut down fine. Then, one evening I used Acme to take some notes.
> At about 05:00, I was bragging to a buddy of mine how cool Plan 9 was.
> No sooner than I began to show him Acme's nifty chording, did the rio
> window began filling up with error messages about trouble writing to the
> hard drive. Although rio kept running, the file system hung. Not being
> able to recover the system, I rebooted it. Well, I tried to. The boot
> process didn't get very far. Ever since then, every time I try to boot
> it, the kernel panics. (Messages below.)
>
> If I knew anything about how the root filesystem was brought online
> during boot, I might be able to debug this myself. Alas, I have no idea
> which programs are generating which parts of this output, so I really
> have no angle from which to attack the problem. Any ideas? The P9
> partition is the third primary partition on the disk. I'm fairly sure
> the HDD is good, because I tested it before the install, and have
> installed and run Linux on the same HDD without incident. Although I
> started and shut down Plan 9 a number of times (just to test its range
> of motion), this crash occurred the first time I tried saving any data
> to the file system (a text file under /usr/glenda/).
>
> Any help would be much appreciated!
>
> Here's what the console shows (typed by hand, may be typos):
>
> root is from (tcp, local)[local!#S/sdC0/fossil]:
> user[none]: glenda
> time...
> venti...2011/0318 00:24:56 venti: conf.../boot/venti: mem 6,252,817 bcmem
12,505,634 icmem 18,758,451...httpd tcp!127.1!8000...init...icache
18,758,451 bytes = 293,101 entries; 4 scache
> sync...2011/0318 00:24:56 arenas00: indexing 45372 clumps...
> announce tcp!127.1!17034...serving.
> fossil(#S/sdC0/fossil)...fsys: dialing venti at tcp!127.1!17034
> /boot/fossil: labelUnpack: bad label: 0xcb 0xb2 0x3d22d532 0x8cacb517
0xfc0d00cb
> /boot/fossil: fsOpen error
> fsOpen: corrupted block label
> fsys main open -c 3000: fsOpen: corrupted block label
> fsys: 'main' not open
> fsys main snaptime -s 60 -a 0500 -t 2880: fsys 'main' not open
> version...boot: mount /: fsys: 'main' not open
> panic: boot process died: unknown
> panic: boot process died: unknown
> dumpstack disabled
> cpu0: exiting
>
> --
> +---------------------------------------------------------------+
> |E-Mail: smiley at zenzebra.mv.com PGP key ID: BC549F8B|
> |Fingerprint: 9329 DB4A 30F5 6EDA D2BA 3489 DAB7 555A BC54 9F8B|
> +---------------------------------------------------------------+
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.9fans.net/private/9fans/attachments/20110317/800f3485/attachment.html>

[9fans] New venti install won't boot after 05:00 crash

[ron minnich]

/boot/fossil: labelUnpack: bad label: 0xcb 0xb2 0x3d22d532 0x8cacb517 0xfc0d00cb
/boot/fossil: fsOpen error
fsOpen: corrupted block label
fsys main open  -c 3000: fsOpen: corrupted block label

well, I don't have good words for you :-(

I think the sequence of failures here -- log messages of write errors,
then a corrupted block label -- point to a hardware problem. You might
recover with a reinstall and you might even recover if you know the
root score of venti and boot the install cd, but in any event I think
it's a bad day and you're going to need to think about a new hard
drive or motherboard.

Sorry.

ron

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

> well, I don't have good words for you :-(
> 
> I think the sequence of failures here -- log messages of write errors,
> then a corrupted block label -- point to a hardware problem. You might
> recover with a reinstall and you might even recover if you know the
> root score of venti and boot the install cd, but in any event I think
> it's a bad day and you're going to need to think about a new hard
> drive or motherboard.

there's an easy way to test this.  since you can boot
from the live cd, just reboot into the live cd and
the open another window.  turn dma on
	; cd /dev/sdC0
	; echo dmaon>ctl
then dd the partition you wish, or the whole disk
(data) with
	; dd -if $partition -of /dev/null -bs 512k &
you can check on your progress with
	; cat /proc/`{ps|grep dd|awk '{print $2}'}^/fd
dd may or may not choke when it hits the end of the
partition, so check the error and the blocks in carefully.
you may be interested in what the console has to say.
i forget if the installer has a window for this, but you
can always view the console log with
	; cat /dev/kmesg

- erik

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

John Floren <john at jfloren.net> writes:

> After the initial install, did you reboot pretty soon? Venti wants to spend
> some time writing an initial image after the first boot, it seems, and if
> you interrupt that you're in trouble

Whenever possible, I use fshalt when shutting down.  (The only excepting
being lock-ups.)  Is it possible that halting the file system with
fshalt gave venti insufficient opportunity to write its initial image?

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

erik quanstrom <quanstro at quanstro.net> writes:

>> it's a bad day and you're going to need to think about a new hard
>> drive or motherboard.
>
> there's an easy way to test this.  since you can boot
> from the live cd, just reboot into the live cd and

I booted the install CD, turned on DMA, dd'd the disk to /dev/null.  No
errors appeared on the console or /dev/kmesg.  I then reinstalled Plan 9
using the default "prep" for fossil+venti.

I rebooted, Plan 9 loaded, and I let it sit.  System load was high,
which I presume was venti doing its thing in the background.  After a
few minutes, system load dropped to zero, and the following messages
appeared on the screen, just like happened the previous time.  (Only
this time, I wasn't bragging about the OS when it crashed. ;)

I wrote ABSOLUTELY nothing to the fossil fs, or changed any file
attributes, after booting.  I just let it sit there peacefully and do
its thing.  Alas, it seems to have failed in more or less the same way.

(Hm.  After a reboot, Plan 9 loads again.  However, it immediately
starts generating those errors, again, with score 0x0000baf2, score
0x0000bafc, score 0x0000bb16, and score 0x0000bb17.  Typing "fshalt" at
the rc prompt returns "fshalt: error reading block 0x0000595e".)

Here's what the console shows (typed by hand, may be typos):

fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000bb17: date Fri Mar 18 15:08:43 EDT 2011 part=data block 47895: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000003b: date Fri Mar 18 15:08:43 EDT 2011 part=data block 59: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000003e: date Fri Mar 18 15:08:43 EDT 2011 part=data block 62: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x00000049: date Fri Mar 18 15:08:43 EDT 2011 part=data block 73: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000004a: date Fri Mar 18 15:08:43 EDT 2011 part=data block 74: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000004b: date Fri Mar 18 15:08:44 EDT 2011 part=data block 75: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000004c: date Fri Mar 18 15:08:44 EDT 2011 part=data block 76: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000004d: date Fri Mar 18 15:08:44 EDT 2011 part=data block 77: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000004e: date Fri Mar 18 15:08:44 EDT 2011 part=data block 78: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x0000004f: date Fri Mar 18 15:08:45 EDT 2011 part=data block 79: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x00000050: date Fri Mar 18 15:08:45 EDT 2011 part=data block 80: i/o error
fossil: diskWriteRaw failed: /dev/sdC0/fossil: score 0x00000051: date Fri Mar 18 15:08:45 EDT 2011 part=data block 81: i/o error
2011/0318 19:10:37 err 4: write /dev/sdC0/arenas offset 0x7aa6000 count 8192 buf 3840000 returned -1: i/o error
/boot/venti: writeproc: part /dev/sdC0/arenas addr 0x7aa6000: write error: write /dev/sdC0/arenas offset 0x7aa6000 count 8192 buf 3840000 returned -1: i/o error
2011/0318 19:10:37 err 4: write /dev/sdC0/arenas offset 0x1ffa4000 count 8192 buf 39e8000 returned -1: i/o error
/boot/venti: writeproc: part /dev/sdC0/arenas addr 0x1ffa4000: write error: write /dev/sdC0/arenas offset 0x1ffa4000 count 8192 buf 39e8000 returned -1: i/o error
2011/0318 19:10:37 err 4: write /dev/sdC0/arenas offset 0x200c0000 count 8192 buf 4024000 returned -1: i/o error
/boot/venti: writeproc: part /dev/sdC0/arenas addr 0x200c0000: write error: write /dev/sdC0/arenas offset 0x200c0000 count 8192 buf 4024000 returned -1: i/o error

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[Jacob Todd]

Did you zero the plan 9 partition first? I recall having a problem like this
when installing over an existing plan 9 installation,  all i had to do was
zero that partition and everything worked fine after that.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.9fans.net/private/9fans/attachments/20110318/85cf5dad/attachment.html>

[9fans] New venti install won't boot after 05:00 crash

[rod at hemiola.co.uk]

I have seen similar symptoms. In the end, I narrowed my problems down
to a very particular combination of hard drive and laptop. 

The installation would complete fine, but I would get i/o errors
running plan9 from the (pata) hard disk. I tried with and without dma
and rwm. "dd" read and write tests run from the installation disk all
proved to be ok. Stand-alone hard disk and memory diagnostics didn't
show up any hardware errors. Another hard drive worked fine with the
same laptop and, bizzarely, the hard drive worked fine with plan9 in a
different laptop.  Debian and windows installed and ran fine.
Interestingly, I had another laptop of the same model (inspiron 3500)
which also failed identically. 

I concluded that there was most likely something about the timing in
the ata driver that tickled the drive/interface combination.

Rod

[9fans] New venti install won't boot after 05:00 crash

[John Floren]

On Fri, Mar 18, 2011 at 9:12 AM, Jacob Todd <jaketodd422 at gmail.com> wrote:
> Did you zero the plan 9 partition first? I recall having a problem like this
> when installing over an existing plan 9 installation,? all i had to do was
> zero that partition and everything worked fine after that.

Yes, be sure to try this. A couple years ago this bit me... at least
at that time, if you re-installed over an existing Plan 9 filesystem,
it wouldn't actually re-format the partition. So I kept re-installing
and finding my same old configuration files and stuff in my home
directory.


John

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

try a write test.  also, you may want to check out how venti/fossil
have repartitioned their play areas.  i believe i've seen them mess
up the size calculations.

- erik

[9fans] New venti install won't boot after 05:00 crash

[Steve Simon]

Another thought - did you let the installer automatically parttition your disk
or did you do it manually - if you overlap partitions or extend one of the
partitions off the end of the disk you can get similar nasty results.

just a random guess.

-Steve

[9fans] New venti install won't boot after 05:00 crash

[ron minnich]

I don't know guys, these sure look like disk errors to me ...

ron

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

On Fri Mar 18 20:03:00 EDT 2011, rminnich at gmail.com wrote:
> I don't know guys, these sure look like disk errors to me ...

no need to guess, fill the partitions with 0s with dd.  if that
works, then we know it's a bug in venti.

- erik

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

On Fri Mar 18 21:08:09 EDT 2011, quanstro at quanstro.net wrote:
> On Fri Mar 18 20:03:00 EDT 2011, rminnich at gmail.com wrote:
> > I don't know guys, these sure look like disk errors to me ...
> 
> no need to guess, fill the partitions with 0s with dd.  if that
> works, then we know it's a bug in venti.

more detail.

i think it's reasonable to suspect that something's going wrong
in user land.  there are about 10 different ways to get Eio from
devsd.  one is that for writes, sdbio() returns Eio if the block is
not within bounds.

also, at least the 9atom version of the ide (or, sdata, sic) prints this
if an actual ata error occurs

		if(drive->status & Err){
			qunlock(ctlr);
print("atagenio: %llud:%d\n", lba, drive->count);
[...]

and dumps the ata state to the console if the i/o never returns.

- erik

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

rod at hemiola.co.uk writes:

> I have seen similar symptoms. In the end, I narrowed my problems down
> to a very particular combination of hard drive and laptop. 

Interesting.  The system that's misbehaving is a laptop, as well.  Its
specs:

IBM ThinkPad T23
ATA ST94813A, 78140160 * 512 byte sectors

00:00.0 Host bridge: Intel Corporation 82830 830 Chipset Host Bridge (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 42)
00:1f.0 ISA bridge: Intel Corporation 82801CAM ISA Bridge (LPC) (rev 02)
00:1f.1 IDE interface: Intel Corporation 82801CAM IDE U100 Controller (rev 02)

erik quanstrom <quanstro at labs.coraid.com> writes:

> try a write test.  also, you may want to check out how venti/fossil
> have repartitioned their play areas.  i believe i've seen them mess
> up the size calculations.

I wiped the entire Plan 9 partition with "dd if=/dev/zero of=/dev/sda3",
which completed without any errors.  I then booted the install CD and
reinstalled.  (I'm getting pretty good at that ;) On the first boot, I
let it sit unmolested.  When the system load trailed off, the screen
began filling with write errors, and the filesystem hung, just like the
previous two install attempts.

Here's a pastebin to a full transcript of the install:

http://pastebin.com/cXpegrhH

Details of the venti layout are in the transcript.

Here are the console messages displayed by the installer:

Fri Mar 18 21:11:24 Installation process started
% fossil/fossil -c . /env/fossilconf
prompt: % mount -c /srv/fossil /n/newfs
fsys main create /active/adm adm sys d775
prompt: fsys main create /active/adm/users adm sys 664
prompt: uname upas :upas
prompt: users -w
	nuser 6 len 96
prompt: fsys main create /active/dist sys sys d775
prompt: fsys main create /active/dist/replica sys sys d775
prompt: fsys main create /active/dist/replica/client sys sys d775
prompt: fsys main create /active/dist/replica/client/plan9.db sys sys 664
prompt: fsys main create /active/dist/replica/client/plan9.log sys sys a664
prompt: % 9660srv
% mount /srv/9660 /n/distmedia /dev/sdD0/data
Initializing Plan 9 FAT partition.
% cp /n/newfs/386/9load /n/9fat/9load
% cp /n/newfs/386/9pcf /n/9fat/9pcf
Setting Plan 9 partition active.

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

> erik quanstrom <quanstro at labs.coraid.com> writes:
> 
> > try a write test.  also, you may want to check out how venti/fossil
> > have repartitioned their play areas.  i believe i've seen them mess
> > up the size calculations.
> 
> I wiped the entire Plan 9 partition with "dd if=/dev/zero of=/dev/sda3",
> which completed without any errors.  I then booted the install CD and
> reinstalled.  (I'm getting pretty good at that ;) On the first boot, I
> let it sit unmolested.  When the system load trailed off, the screen
> began filling with write errors, and the filesystem hung, just like the
> previous two install attempts.

i have a very hard time believing that this can be
explained by a hardware problem.  i've seen a lot of
crazy disk behavior, but i have not seen lbas that
are writable only by dd, and not by other programs. :-)

i think this is our menu of believable options:

1.  ide driver problem
2.  venti problem
3.  fossil problem

the most likely problem is that venti, fossil or both
are writing outside their alloted partitions.  the 64-bit
question is, which partitions.  if the partition sizes you've
got don't look silly, it may be hard to track this down.

the easiest thing to do might be to try a fossil-only install.
it's not ideal, but maybe it's worth a shot.

- erik

[9fans] New venti install won't boot after 05:00 crash

[cinap_lenrek at gmx.de]

check the output of cat /dev/sdC0/ctl for lba48always.

i have it enabled on my t23 because i got i/o erros (on a particular
block number that i forgot...  it was reproducable) if its off.

thats what i have in my termrc:

echo dma on >/dev/sdC0/ctl
echo lba48always on >/dev/sdC0/ctl

the machine runs with 64 bit cwfs with 16K blocksize.

--
cinap


-------------- next part --------------
An embedded message was scrubbed...
From: smiley at zenzebra.mv.com
Subject: Re: [9fans] New venti install won't boot after 05:00 crash
Date: Sat, 19 Mar 2011 02:15:33 +0000
Size: 5196
URL: <http://mail.9fans.net/private/9fans/attachments/20110319/73f79cc0/attachment.mht>

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

On Fri Mar 18 22:48:04 EDT 2011, cinap_lenrek at gmx.de wrote:

> check the output of cat /dev/sdC0/ctl for lba48always.
> 
> i have it enabled on my t23 because i got i/o erros (on a particular
> block number that i forgot...  it was reproducable) if its off.

i don't really think this is the problem, since a dd on the whole
disk worked.

the reason for your error is there's an obi wan in the sources code.
the reason for the misunderstanding is the wierd way ata reports
the size of the disk.  i submitted a patch with this long ago.

but perhaps we should just decide that all disk drives alive today
can do 48 bit addressing, and just do that.

enum{
	Last28	= (1<<28) - 1 - 1,
};

static int
atageniostart(Drive* drive, uvlong lba)
{
	Ctlr *ctlr;
	uchar cmd;
	int as, c, cmdport, ctlport, h, len, s, use48;

	use48 = 0;
	if((drive->flags&Lba48always) || lba > Last28 || drive->count > 256){

- erik

[9fans] New venti install won't boot after 05:00 crash

[ron minnich]

On Fri, Mar 18, 2011 at 7:53 PM, erik quanstrom <quanstro at quanstro.net> wrote:

> i don't really think this is the problem, since a dd on the whole
> disk worked.

If you have a bad bit of memory it can result in exactly these sorts
of problems. Seen on a T23 I used to have, esp. when it decided to get
too warm. There are lots of ways that a dd on a disk can work and a
venti can fail, all due to hardware issues, seen that too.

That said I trust your opinion more than mine :-)

and it's always better if it's a simple fix. Good luck!

ron

[9fans] New venti install won't boot after 05:00 crash

[cinap_lenrek at gmx.de]

makes sense... this was a long time ago... :)

--
cinap
-------------- next part --------------
An embedded message was scrubbed...
From: erik quanstrom <quanstro at quanstro.net>
Subject: Re: [9fans] New venti install won't boot after 05:00 crash
Date: Fri, 18 Mar 2011 22:53:32 -0400
Size: 2910
URL: <http://mail.9fans.net/private/9fans/attachments/20110319/1ff2c0de/attachment.mht>

[9fans] New venti install won't boot after 05:00 crash

[cinap_lenrek at gmx.de]

lie! T23 never fails! ;-)

--
cinap
-------------- next part --------------
An embedded message was scrubbed...
From: ron minnich <rminnich at gmail.com>
Subject: Re: [9fans] New venti install won't boot after 05:00 crash
Date: Fri, 18 Mar 2011 20:27:18 -0700
Size: 3615
URL: <http://mail.9fans.net/private/9fans/attachments/20110319/d54f5d55/attachment.mht>

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

erik quanstrom <quanstro at quanstro.net> writes:

> On Fri Mar 18 22:48:04 EDT 2011, cinap_lenrek at gmx.de wrote:
>
>> check the output of cat /dev/sdC0/ctl for lba48always.
>> 
>> i have it enabled on my t23 because i got i/o erros (on a particular
>> block number that i forgot...  it was reproducable) if its off.
>
> i don't really think this is the problem, since a dd on the whole
> disk worked.

In the interest of full disclosure, that "dd" was done under Linux.  It
was a "dd if=" rather than a "dd -if".  However, I just re-ran the dd
from the Plan 9 installer, using the Plan 9 dd to read the entire
partition and zero the entire partition.  Neither the read nor write
returned any errors.  This is with NO lba48always in cat /dev/sdC0/ctl.

My next step will be to attempt a fossil-only install.  If that doesn't
generate these kinds of errors, at least we'll know the problem is
venti's work.

Has anyone looked at the "prep" layout in the install transcript I
bastebin'd?  Does it look sane?

I really wish I understood how these file servers worked... I would be
able to debug this by myself.  Alas, Plan 9 is a network operating
system.  So, I suppose it's only natural that troubleshooting be
networked, too.  ;)

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

On Sat Mar 19 10:22:11 EDT 2011, smiley at zenzebra.mv.com wrote:
> In the interest of full disclosure, that "dd" was done under Linux.  It
> was a "dd if=" rather than a "dd -if".  However, I just re-ran the dd
> from the Plan 9 installer, using the Plan 9 dd to read the entire
> partition and zero the entire partition.  Neither the read nor write
> returned any errors.  This is with NO lba48always in cat /dev/sdC0/ctl.

> Has anyone looked at the "prep" layout in the install transcript I
> bastebin'd?  Does it look sane?

at first glance, it does.  if the lba48 problem is it, then it won't kick
in until sector 268435454 + 1.  you didn't send the contents of /dev/sdC0/ctl,
so it's hard to know where plan 9 partitions ended up.  

- erik

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

smiley at zenzebra.mv.com writes:

> My next step will be to attempt a fossil-only install.  If that doesn't
> generate these kinds of errors, at least we'll know the problem is
> venti's work.

Well, after zeroing the partition, I installed a fossil-only system.
Upon first boot, the system begins giving me similar i/o errors, and the
file system hangs.

This is odd, because I did "dd" reads and writes of the entire Plan 9
partition FROM the Plan 9 installer.  The partition can be read/written
from the installer, but not from the installed system.  So... what
difference between the install system and the installed system could be
responsible for this?  (Venti is no longer part of the equation.)

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

> This is odd, because I did "dd" reads and writes of the entire Plan 9
> partition FROM the Plan 9 installer.  The partition can be read/written
> from the installer, but not from the installed system.  So... what
> difference between the install system and the installed system could be
> responsible for this?  (Venti is no longer part of the equation.)

there are a number of differences.  here are a few that spring
to mind as important
- the kernel in the cd is different,
- it uses a different plan9.ini file, and
- dma is setup after booting.

it would be good to review the plan9.ini file generated.
i don't remember what they look like.

also, i remember a year or so ago, someone had a system with a ide
device that didn't take kindly to pio commands.  unfortunately, when
you boot the system, fossil does i/o before dma could be turned on.
(since the command to turn it on is sitting inside the fossil.)  perhaps you
could try putting ftp://ftp.quanstro.net/other/9pcf.gz in your 9fat,
replacing your current kernel, and setting the plan9.ini varaible "*sdC0dma=on".

- erik

[9fans] New venti install won't boot after 05:00 crash

[cinap_lenrek at gmx.de]

how did the lba48 bug affect the translation of linear byte offsets
exactly?

--
cinap
-------------- next part --------------
An embedded message was scrubbed...
From: erik quanstrom <quanstro at quanstro.net>
Subject: Re: [9fans] New venti install won't boot after 05:00 crash
Date: Sat, 19 Mar 2011 10:32:21 -0400
Size: 2828
URL: <http://mail.9fans.net/private/9fans/attachments/20110319/15a80232/attachment.mht>

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

On Sat Mar 19 12:36:31 EDT 2011, cinap_lenrek at gmx.de wrote:

> how did the lba48 bug affect the translation of linear byte offsets
> exactly?

the fix means that the transition to 48-bit addressing happens
1 sector sooner.

the reason for this is when the size of a device is reported, the
max lba + 1 is reported, not the maximum lba.  thus a 28-bit
device reporting fffffff would not be able to address sector
fffffff; the last addressable sector would be ffffffe.

- erik

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

erik quanstrom <quanstro at quanstro.net> writes:

> there are a number of differences.  here are a few that spring
> to mind as important
> - the kernel in the cd is different,
> - it uses a different plan9.ini file, and
> - dma is setup after booting.

> could try putting ftp://ftp.quanstro.net/other/9pcf.gz in your 9fat,
> replacing your current kernel, and setting the plan9.ini varaible
> "*sdC0dma=on".

That did it.  To test the theory that PIO was the problematic variable,
I tried booting the installer with DMA *off*.  Neither the fossil or
fossil+venti installs would complete without reporting HDD errors.  So,
I plopped your 9pcf.gz on 9fat, and set *sdC0dma=on.  Now, a
fossil+venti is running without any apparent problems.  :) :)

As a workaround, I tried putting "echo dma on > /dev/sdC0/ctl" in
termrc.  This was sufficient to keep a fossil-only install from failing.
But, even *with* this workaround, a fossil+venti install would still
return i/o errors.  That's probably because fossil calls venti before
init calls termrc.  Even wrapping init in a shell script probably
wouldn't get DMA turned on soon enough to make venti happy, but your
kernel did the trick.

Here's the workaround I used:

  # install Plan 9
  user[none]: sys
  # rio fails to start
  echo dma on > /dev/sdC0/ctl # run this command immediately!
  cd /rc/bin
  mv termrc termrc.was
  cat > termrc
  #!/bin/rc
  echo dma on > /dev/sdC0/ctl
  ^D
  cat termrc.was >> termrc
  diff termrc.was termrc
  # confirm the edit before removing termrc.was!
  rm termrc.was
  fshalt

That seemed to work for a fossil-only system.  But that 9pcf.gz was
necessary to make venti happy.

Reading fossil(4), fossilcons(8), and boot(8) really helped to get a
good understanding of how the file system is set up at boot.  venti(6)
and venti(8) provided a good overview of venti.  One gotcha, though: you
have to read con(1) to find out you can quit fossilcons with control-\

Thanks for the help!

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

> As a workaround, I tried putting "echo dma on > /dev/sdC0/ctl" in
> termrc.  This was sufficient to keep a fossil-only install from failing.
> But, even *with* this workaround, a fossil+venti install would still
> return i/o errors.  That's probably because fossil calls venti before
> init calls termrc.  Even wrapping init in a shell script probably
> wouldn't get DMA turned on soon enough to make venti happy, but your
> kernel did the trick.

great work.

i do want to point out that your trick to get the fossil-only install
going is not going to be reliable.  it relies on the boot process winning
a race that it is likely to lose; you'll need to run the 9atom kernel
regardless.

does anything think that it's a mistake to default dma on?

- erik

[9fans] New venti install won't boot after 05:00 crash

[Lucio De Re]

> does anything think that it's a mistake to default dma on?

I have a lot of old kit around and can't for the hell of me figure out
which drives do need and which ones don't want DMA, occasionally
losing if nothing else a lot of time and effort in repairing a bad
assumption.  Having a kernel instruction can only be very useful;
setting an arbitrary initial condition isn't really that helpful
either way.

The default, of course, would have to be ON if most hardware is to be
taken into account.  I'm not sure, given that old drives had unlimited
lifespans and new drives only last a year or so, who wins that
competition :-) Probably the motherboards: the old ones just can't
compete.

++L

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

> I have a lot of old kit around and can't for the hell of me figure out
> which drives do need and which ones don't want DMA, occasionally

i thought that dma problems were almost always controller/cable or master/slave
related?  almost all the device i deal with are sata these days.  and before
that, i stuck to scsi.

in any event, 
-  there's a knob because i was worried about breaking old stuff.
-  it's likely the ide driver, rather than hardware, causes this incompatability.

smiley, did the kernel with *sdC0dma support print anything different?

- erik

[9fans] New venti install won't boot after 05:00 crash

[Lucio De Re]

> i thought that dma problems were almost always controller/cable or master/slave
> related?  almost all the device i deal with are sata these days.  and before
> that, i stuck to scsi.
> 
Sure.  But you at Coraid and Geoff at Bell Labs are professionals, I'm
assuming (happy to be corrected) that most of us here are fumbling
amateurs and unlikely to invest our priciest resources on Plan 9
equipment.  Although, thinking about it...

> in any event, 
> -  there's a knob because i was worried about breaking old stuff.
> -  it's likely the ide driver, rather than hardware, causes this incompatability.

Good stuff.  Keep the knob and set it on by default.  Then get Geoff
and Jim to catch up with your great work, it's about time.

++L

[9fans] New venti install won't boot after 05:00 crash

[smiley at zenzebra.mv.com]

erik quanstrom <quanstro at quanstro.net> writes:

> i thought that dma problems were almost always controller/cable or master/slave
> related?  almost all the device i deal with are sata these days.  and before
> that, i stuck to scsi.

> smiley, did the kernel with *sdC0dma support print anything different?

Yes, it did.  I noticed that the DMA kernel prints "nomp: no mp
structure" at bootup.  But I didn't save kmesg from the failed installs,
so I can't give you any more diffs than that.

-- 
+---------------------------------------------------------------+
|E-Mail: smiley at zenzebra.mv.com             PGP key ID: BC549F8B|
|Fingerprint: 9329 DB4A 30F5 6EDA D2BA  3489 DAB7 555A BC54 9F8B|
+---------------------------------------------------------------+

[9fans] New venti install won't boot after 05:00 crash

[erik quanstrom]

> > smiley, did the kernel with *sdC0dma support print anything different?
> 
> Yes, it did.  I noticed that the DMA kernel prints "nomp: no mp
> structure" at bootup.  

that's related to how interrupts are delivered.  it doesn't have anything
directly to do with ide.

> But I didn't save kmesg from the failed installs,
> so I can't give you any more diffs than that.

okay,  no biggie.  but i guess the reason for the failure
will remain a mystery for now.  i'll chalk it up as, likely
a driver falure.

- erik