[9fans] oh, no! (again)

[9fans] oh, no! (again)

[cej]

Sadly, I was too optimistic when SATA install succeeded... :-(
This is a report of error messages I've got while booting new system on (and from) sdE0:

sdiahci: drive 0 in state ready after 0 resets   # SATA HD with Plan 9 system on it
sdiahci: drive 1 won't come up; in state new after 10 resets   # SATA DVD
sdE1 waitready: [new] task=50 sstat=113
sdE1 waitready: [new] task=50 sstat=113
sdE1 waitready: [new] task=50 sstat=113
sdE1 waitready: [new] task=50 sstat=113
sdE1 not responding; offline
sdiahci: drive 2 won't come up; in state portreset after 10 resets
sdiahci: drive 3 won't come up; in state portreset after 10 resets
sdiahci: drive 0 in state missing after 0 resets   # strange!
sdiahci: drive 1 in state missing after 0 resets   # strange!
.
1. 1
2. 1
3. none of the above
*nobiosload: 1
[snip]
user[none]: glenda
sdE0: LLBA blah blah ...
sdE1: LBA 0 sectors
time...
fossil(#S/sdE0/fossil)...version...time
init: starting /bin/rc
sdEO: i/o error d0 @206,672
fossil: diskReadRaw failed: /dev/sdE0/fossil: score 0x0000005e: part=label block 94: i/o error
sdEO: i/o error d0 @ 3,707,248
fossil: diskReadRaw failed: /dev/sdE0/fossil: score 0x000029a4: part=data block c0660: i/o error
aux/vga: error reading block 0x000029a4
rio: can't open display: initdisplay: /dev/draw/new: no frame buffer
init: rc exit status: rio 29: display open
init: starting /bin/rc
==

Other time, rio comes up, with acme running, but:
term% 9fat:
sdE0: i/o error d0 @ 4,317,856   # the number after @ changes when cmd is repeated
fossil: diskReadRaw failed: /dev/sdE0/fossil: score 0x0000beb7: part=data block 48823: i/o error   # the numbers after 'score' and 'block' change when cmd is repeated

Sorry for being so verbose, but I can't realize which parts are not significant,
thanks, wishing a great day,

++pac.

Re: [9fans] oh, no! (again)

[erik quanstrom]

i'll be working on this today.  you are unfortunately running
and old kernel which is causing trouble.  (as noted by fernanbolando.)
this is my fault for building a bogus iso.

- erik

Re: [9fans] oh, no! (again)

[erik quanstrom]

> sdiahci: drive 0 in state ready after 0 resets   # SATA HD with Plan 9 system on it

that's good.  i hope i've corrected the debug prints.
in this case drive n is sdEn.

> sdiahci: drive 1 won't come up; in state new after 10 resets   # SATA DVD
> sdE1 waitready: [new] task=50 sstat=113
> sdE1 waitready: [new] task=50 sstat=113
> sdE1 waitready: [new] task=50 sstat=113
> sdE1 waitready: [new] task=50 sstat=113
> sdE1 not responding; offline

unfortunately, task=50 and sstat=113 are what we want, but the
old 9load driver just isn't getting the picture.

> sdiahci: drive 2 won't come up; in state portreset after 10 resets
> sdiahci: drive 3 won't come up; in state portreset after 10 resets

this is also bogus.  there's no drive to reset.

> sdiahci: drive 0 in state missing after 0 resets   # strange!
> sdiahci: drive 1 in state missing after 0 resets   # strange!

bug.  probablly a drive power mgmt problem.  was this a seperate boot?

also,
> init: starting /bin/rc
> sdE0: i/o error d0 @206,672

that is Eicrc | Eunc | Eidnf =
	interface crc error,
	uncorrectable read error
	no user-accessable sector.

this is a jmicron bug.  i haven't yet had in my possession a jmicron
controller, so this will be interesting, but the new driver does have
code to work around this very issue.

- erik

Re: [9fans] oh, no! (again)

[cej]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]


First of all, thank you very much for your work, Eric ! 


>> sdiahci: drive 0 in state missing after 0 resets   # strange!
>> sdiahci: drive 1 in state missing after 0 resets   # strange!

> bug.  probablly a drive power mgmt problem.  was this a seperate boot?

No. 


> this is a jmicron bug.  i haven't yet had in my possession a jmicron
> controller, so this will be interesting, but the new driver does have
> code to work around this very issue.

My fault. I forwarded the 'Plan 9 Hardware Specification Page' to the computer enterprise who assembled my comp. Apparently, he ignored the 'Chipset' paragraph.
Thank you very much for your effort, indeed,

truly,
Peter, aka ++pac.

- erik



[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 2660 bytes --]

Re: [9fans] oh, no! (again)

[erik quanstrom]

>
> First of all, thank you very much for your work, Eric !

you're welcome.  unfortunately, so far we haven't made any
progress.

> > this is a jmicron bug.  i haven't yet had in my possession a jmicron
> > controller, so this will be interesting, but the new driver does have
> > code to work around this very issue.
>
> My fault. I forwarded the 'Plan 9 Hardware Specification Page' to the computer enterprise who assembled my comp. Apparently, he ignored the 'Chipset' paragraph.
> Thank you very much for your effort, indeed,

i've verified that i see exactly what you see on my intel atom machine.
including the d0 errors, so there's something reazlly hinky going on.

after replacing the kernel and 9load in 9fat:, however, my machine
boots up just fine.

i don't know where it's getting the 9pcf that it's putting in 9fat.  it
doesn't appear to be the same as /386/9pcf from the cd.  (there's
only one.)

as soon as i figure that out, this will probablly all work.

- erik

Re: [9fans] oh, no! (again)

[erik quanstrom]

>
> i don't know where it's getting the 9pcf that it's putting in 9fat.  it
> doesn't appear to be the same as /386/9pcf from the cd.  (there's
> only one.)
>
> as soon as i figure that out, this will probablly all work.

okay.  the cd was rolled correctly, but i discovered a potential
source of confusion.  evidently the installer didn't want to
overwrite the bad 9pcf and 9load that were already in 9fat.
so download a new copy of 9atom.tar.bz2 and when you
start the installer, make sure to zero out 9fat at a minimum.
to do this
	- open a new window
	- dd -if /dev/zero -of /dev/sdXX/9fat -bs 4k

a further note, if you rebuild your kernel, it won't work.
installing the following contrib packages is required
	quanstro/fis		(build and parse sata fises)
	quanstro/sd		(new sd stuff)
	quanstro/nvram		(so you can find nvram)
	quanstro/9load-e820	(to rebuild 9load)

for most atom-based machines with 8169s you will need
this as well.

	quanstro/8169

comment out the edev->maxtu line.  that's for jumbo frames
support for aoe.  you could apply the changes to the dist
driver as well, but unfortunately the distributed driver does
not handle unexpectly large packet reception.  (crash)

also you may wish to enable support for the new sd stuff.
	quanstro/atazz		(send raw ata commands)
	quanstro/smart		(smart monitor)

there are a few things in the kernel i've built that don't fit
in dist packages very well.
	PAT
	jumbo ethernet frames
	kernel cec server		(console-over-ethernet)
	/dev/cputemp
	/dev/nomp		(mptables when not enabled)

sorry about that.

finally, would anyone who has success or failure please
send me a note off-list including lspci output and what
happens when you try the official distribution.

- erik

Re: [9fans] oh, no! (again)

[David Leimbach]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On Tue, Aug 11, 2009 at 10:57 AM, erik quanstrom <quanstro@quanstro.net>wrote:

> >
> > i don't know where it's getting the 9pcf that it's putting in 9fat.  it
> > doesn't appear to be the same as /386/9pcf from the cd.  (there's
> > only one.)
> >
> > as soon as i figure that out, this will probablly all work.
>
> okay.  the cd was rolled correctly, but i discovered a potential
> source of confusion.  evidently the installer didn't want to
> overwrite the bad 9pcf and 9load that were already in 9fat.
> so download a new copy of 9atom.tar.bz2 and when you
> start the installer, make sure to zero out 9fat at a minimum.
> to do this
>        - open a new window
>        - dd -if /dev/zero -of /dev/sdXX/9fat -bs 4k
>

When I download this ISO, it gets to what appears to be the last byte then
aborts.

[-- Attachment #2: Type: text/html, Size: 1210 bytes --]

Re: [9fans] oh, no! (again)

[erik quanstrom]

>
> When I download this ISO, it gets to what appears to be the last byte then
> aborts.

i tried this and it works for me.  in the interest of time, and
not further overloading my dsl line, i downloaded over the
local interface, and not going out through the intertubes.
if you were downloading when i swapped files out
(Tue Aug 11 16:34:05 GMT 2009  (12:34:05 EDT))
you may have gotten bitten by that.  i have

md5sum	c6952bc573d9800f7d63d5a017918094	9atom.iso.bz2
sha1sum 8ae0dc5657a3810845a83c6e94db35ba7ca6e48b	9atom.iso.bz2
; ls -l 9atom.iso.bz2
--rw-r--r-- M 129 quanstro quanstro 88214974 Aug 11 11:15 9atom.iso.bz2

let me know if this problem persists.

- erik

Re: [9fans] oh, no! (again)

[Fernan Bolando]

On Wed, Aug 12, 2009 at 3:25 AM, erik quanstrom<quanstro@quanstro.net> wrote:
>>
>> When I download this ISO, it gets to what appears to be the last byte then
>> aborts.
>
> i tried this and it works for me.  in the interest of time, and
> not further overloading my dsl line, i downloaded over the
> local interface, and not going out through the intertubes.
> if you were downloading when i swapped files out
> (Tue Aug 11 16:34:05 GMT 2009  (12:34:05 EDT))
> you may have gotten bitten by that.  i have
>
> md5sum  c6952bc573d9800f7d63d5a017918094        9atom.iso.bz2
> sha1sum 8ae0dc5657a3810845a83c6e94db35ba7ca6e48b        9atom.iso.bz2
> ; ls -l 9atom.iso.bz2
> --rw-r--r-- M 129 quanstro quanstro 88214974 Aug 11 11:15 9atom.iso.bz2
>
> let me know if this problem persists.
>
the new iso got me a bit further, however after boot it did not see my
sdE0 bootable device

last 2 lines says
Boot devices: fd0 ether0
boot from:


I will investigate further.


-- 
http://www.fernski.com

Re: [9fans] oh, no! (again)

[erik quanstrom]

>
> last 2 lines says
> Boot devices: fd0 ether0
> boot from:
>
>
> I will investigate further.
>

judging from past email, i'm guessing that your ide device
is 27c4, which was missing from 9load.  i put up a corrected cd.
if you can ftpfs from your mostly-installed machine and
get just 9load.bz2, it's in the same directory.

; i=9atom.iso.bz2 sha1sum $i && ls -l $i
cb7bdb9bcaebaf5e54d5a76c672dba52f90eb55a	9atom.iso.bz2
--rw-r--r-- M 176 quanstro quanstro 88211681 Aug 11 17:13 9atom.iso.bz2

- erik

Re: [9fans] oh, no! (again)

[Fernan Bolando]

On Wed, Aug 12, 2009 at 5:17 AM, erik quanstrom<quanstro@quanstro.net> wrote:
>>
>> last 2 lines says
>> Boot devices: fd0 ether0
>> boot from:
>>
>>
>> I will investigate further.
>>
>
> judging from past email, i'm guessing that your ide device
> is 27c4, which was missing from 9load.  i put up a corrected cd.
> if you can ftpfs from your mostly-installed machine and
> get just 9load.bz2, it's in the same directory.
>
> ; i=9atom.iso.bz2 sha1sum $i && ls -l $i
> cb7bdb9bcaebaf5e54d5a76c672dba52f90eb55a        9atom.iso.bz2
> --rw-r--r-- M 176 quanstro quanstro 88211681 Aug 11 17:13 9atom.iso.bz2
>
thanks erik!!!  I can now boot. I am now at the
init starting /bin/rc     stage it seems to stall there. I think its
just the ip/ipconfig issue should be easy to fix.

just a note.  i am getting
ide: caught missed irq   several times shoul i be concerned?

fernan




-- 
http://www.fernski.com

Re: [9fans] oh, no! (again)

[erik quanstrom]

> > judging from past email, i'm guessing that your ide device
> > is 27c4, which was missing from 9load.  i put up a corrected cd.
> > if you can ftpfs from your mostly-installed machine and
> > get just 9load.bz2, it's in the same directory.
> >
> > ; i=9atom.iso.bz2 sha1sum $i && ls -l $i
> > cb7bdb9bcaebaf5e54d5a76c672dba52f90eb55a        9atom.iso.bz2
> > --rw-r--r-- M 176 quanstro quanstro 88211681 Aug 11 17:13 9atom.iso.bz2
> >
> thanks erik!!!  I can now boot. I am now at the
> init starting /bin/rc     stage it seems to stall there. I think its
> just the ip/ipconfig issue should be easy to fix.

> just a note.  i am getting
> ide: caught missed irq   several times shoul i be concerned?

what's going on:
i replaced the sleep in sdata.c with a loop that checks for i/o
completion every second.  i/o should never be outstanding for
1 second, so this will never fire unless your disk is very sick,
or you're missing interrupts.  after 10 missed irqs, it has mercy
on you and checks every 50ms.  this will seem very slow.

the pureists are probablly going to choke at this trick.
i don't blame them.  but on the other hand, i want to give
folks a chance to recover their systems if ide acts up.

this is something to be concerned about.

once you boot up, there are a number of things to try.
/dev/irqalloc for your kernel should have an extra final
entry that counts the number of interrupts on each vector.
i would expect that number would be 0, given your symptoms.

here are a few things you can look at
- is your southbridge recognized?

- you might want to try turning dma on, on the wild
theory that it worked well enough during installation.

- you might want to try turning mp interrupts on.
unfortunately, i forgot to add the nomp device which
allow one to inspect the mp interrupts before trying
them.

failing any of those things, send me your pci output.
this setup should work.

- erik

Re: [9fans] oh, no! (again)

[Fernan Bolando]

On Wed, Aug 12, 2009 at 8:40 PM, erik quanstrom<quanstro@quanstro.net> wrote:
>> > judging from past email, i'm guessing that your ide device
>> > is 27c4, which was missing from 9load.  i put up a corrected cd.
>> > if you can ftpfs from your mostly-installed machine and
>> > get just 9load.bz2, it's in the same directory.
>> >
>> > ; i=9atom.iso.bz2 sha1sum $i && ls -l $i
>> > cb7bdb9bcaebaf5e54d5a76c672dba52f90eb55a        9atom.iso.bz2
>> > --rw-r--r-- M 176 quanstro quanstro 88211681 Aug 11 17:13 9atom.iso.bz2
>> >
>> thanks erik!!!  I can now boot. I am now at the
>> init starting /bin/rc     stage it seems to stall there. I think its
>> just the ip/ipconfig issue should be easy to fix.
>
>> just a note.  i am getting
>> ide: caught missed irq   several times shoul i be concerned?
>
> what's going on:
> i replaced the sleep in sdata.c with a loop that checks for i/o
> completion every second.  i/o should never be outstanding for
> 1 second, so this will never fire unless your disk is very sick,
> or you're missing interrupts.  after 10 missed irqs, it has mercy
> on you and checks every 50ms.  this will seem very slow.
>

Its very very slow hehehe

> the pureists are probablly going to choke at this trick.
> i don't blame them.  but on the other hand, i want to give
> folks a chance to recover their systems if ide acts up.
>
> this is something to be concerned about.
>
> once you boot up, there are a number of things to try.
> /dev/irqalloc for your kernel should have an extra final
> entry that counts the number of interrupts on each vector.
> i would expect that number would be 0, given your symptoms.

term% cat /dev/irqalloc
3 0 debugpt [0]
7 0 mathemu [0]
8 0 doublefault [0]
9 0 mathover [0]
14 0 fault386 [0]
15 0 unexpected [0]
16 0 matherror [0]
32 0 clock [8298344]
33 1 kbd [103]
38 6 floppy [0]
39 7 lpt [29]
42 10 usbehci [0]
42 10 usbuhci [0]
43 11 sdE (ata) [16869]
43 11 usbuhci [16863]
43 11 usbuhci [16863]
43 11 ether0 [16863]
44 12 kbdaux [2400]
46 14 sdC (ata) [23]



>
> here are a few things you can look at
> - is your southbridge recognized?
>
not sure how to check this...

> - you might want to try turning dma on, on the wild
> theory that it worked well enough during installation.

updated /rc/bin/cpurc.local to include dmamode=ask
rebooting now.

> - you might want to try turning mp interrupts on.
> unfortunately, i forgot to add the nomp device which
> allow one to inspect the mp interrupts before trying
> them.
plan9.ini already shows *nomp=1
so I guess it's a default.

> failing any of those things, send me your pci output.
> this setup should work.
dont have lspci will install after trying the dmamode.

thanks for the patience.



-- 
http://www.fernski.com

Re: [9fans] oh, no! (again)

[Fernan Bolando]

On Wed, Aug 12, 2009 at 1:57 AM, erik quanstrom<quanstro@quanstro.net> wrote:

>
> a further note, if you rebuild your kernel, it won't work.
> installing the following contrib packages is required
>        quanstro/fis            (build and parse sata fises)
>        quanstro/sd             (new sd stuff)
>        quanstro/nvram          (so you can find nvram)
>        quanstro/9load-e820     (to rebuild 9load)

I am suppose to replace /sys/src/boot/pc with /sys/src/boot/pc-e820???

> for most atom-based machines with 8169s you will need
> this as well.
>
>        quanstro/8169
>
> comment out the edev->maxtu line.  that's for jumbo frames
> support for aoe.  you could apply the changes to the dist
> driver as well, but unfortunately the distributed driver does
> not handle unexpectly large packet reception.  (crash)

There is no edev->maxtu in /sys/src/9/pc ether8169.c
there is edev->maxmtu Mtu same thing???

> also you may wish to enable support for the new sd stuff.
>        quanstro/atazz          (send raw ata commands)
>        quanstro/smart          (smart monitor)
>
> there are a few things in the kernel i've built that don't fit
> in dist packages very well.
>        PAT
>        jumbo ethernet frames
>        kernel cec server               (console-over-ethernet)
>        /dev/cputemp
>        /dev/nomp               (mptables when not enabled)
>

 not sure what to do here.


-- 
http://www.fernski.com

Re: [9fans] oh, no! (again)

[erik quanstrom]

On Thu Aug 13 10:16:09 EDT 2009, fernanbolando@mailc.net wrote:
> >        quanstro/9load-e820     (to rebuild 9load)
>
> I am suppose to replace /sys/src/boot/pc with /sys/src/boot/pc-e820???

if you do reinstall, you will need to install from /sys/src/boot/pc-e820
not /sys/src/boot/pc. if you leave /sys/src/boot/pc there or not doesn't matter.

> There is no edev->maxtu in /sys/src/9/pc ether8169.c
> there is edev->maxmtu Mtu same thing???

i don't believe you've installed the version of 8169 in
my contrib.

> > also you may wish to enable support for the new sd stuff.
> >        quanstro/atazz          (send raw ata commands)
> >        quanstro/smart          (smart monitor)

that didn't come out right.  if you're using the new sd stuff, then
these command will work, and you might find them useful.
smart has a better chance of being useful than the rather esoteric
atazz.

> >
> > there are a few things in the kernel i've built that don't fit
> > in dist packages very well.
> >        PAT
> >        jumbo ethernet frames
> >        kernel cec server               (console-over-ethernet)
> >        /dev/cputemp
> >        /dev/nomp               (mptables when not enabled)
> >
>
>  not sure what to do here.

this was just fyi.  these will be differences between the prebuilt
kernel on the iso and one you can rebuild yourself.

i forgot some things like jumbo frames, too.  this is important
for good aoe performance.  i've gotten over 1GB/s of sustaned aoe
traffic on a lowly 1.6ghz xeon 5000 machine over a myricom
10gbe card.  when i get back to working on the myricom stuff,
i will post a link to the exact setup.

again, the iso is just kind of a bandaid.  the contrib packages
are, too.  it's all quite unsatisfactory.  sorry i havn't done a better job.
i'm up for suggestions.

- erik

Re: [9fans] oh, no! (again)

[Fernan Bolando]

On Thu, Aug 13, 2009 at 10:33 PM, erik quanstrom<quanstro@quanstro.net> wrote:
> On Thu Aug 13 10:16:09 EDT 2009, fernanbolando@mailc.net wrote:

>> There is no edev->maxtu in /sys/src/9/pc ether8169.c
>> there is edev->maxmtu Mtu same thing???
>
> i don't believe you've installed the version of 8169 in
> my contrib.
>

term% contrib/install quanstro/8169
8169 is already installed
term% contrib/pull -s sys/src/9/pc 8169
term%

hmm it says already installed, but I cant grep any reference to edev->maxtu




--
http://www.fernski.com

Re: [9fans] oh, no! (again)

[erik quanstrom]

> term% contrib/install quanstro/8169
> 8169 is already installed
> term% contrib/pull -s sys/src/9/pc 8169
> term%
>
> hmm it says already installed, but I cant grep any reference to edev->maxtu

okay, maybe there's not a problem.  if it
just compiles, then don't worry about it.
otherwise, remove the assignment to the
missing structure element.

- erik