[9fans] 9pcf debugging

[9fans] 9pcf debugging

[Lluís Batlle]

I've seen there is rdbfs for debugging the 9pc kernel through a serial port...

The computer the driver fails on doesn't have a serial port. I'd like
to the debug the ethernet driver, so I don't have ethernet. :)

Any ideas, over the common lot-of-print() ?

Thanks in advance.

Re: [9fans] 9pcf debugging

[ron minnich]

[-- Attachment #1: Type: text/plain, Size: 2072 bytes --]

We had this problem too on LinuxBIOS, and it's a real problem when you've
just powered on and have no memory or stack. But some vendors are claiming
they have dropped serial to save money, since we have "the Universal Serial
Bus for serial I/O".

Fortunately, the universal serial bus guys finally figured out, TEN YEARS
LATE, that they had not actually provided  a workable serial bus mechanism
for debugging. Yup, it finally hit them that they somehow forgot to make up
a simple serial output, and that running those serial dongles was a bit more
than most bioses could get, esp. when there is no stack available. It hit
them just about the time somebody said, "I want to remove the serial port,
how do I debug?"

Ah, USB. 24+ chapters and not a single good idea :-)
[ok, maybe one: powered cables are nice]

There is a fix. Well, two fixes. First off, in light of USB's utter
uselessness for debugging, most vendors still put COMA on their boards --
they just don't bring it out to a connector. This is true on almost every
board I have, including a "consumer" board I have here which is advertised
not to have serial!

So open the box and look for COMA. It's usually a 10-pin header, two rows of
5 pins, .1 inch centers, with (if you're lucky) a giant COMA label next to
it.Sometimes they clip one pin so it is oriented.

Option two is really cute, and could only have been a USB invention.
Basically, since there is no easy way to debug the Universal Serial Bus,
they created this kludge called the Debug port.
Here's a talk on it:
http://www.usb.org/developers/presentations/pres0602/john_keys.pdf

note that when a port is used for usb debug, it's disabled for USB. Cute,
huh?

You need to get a debug port dongle, which is so simple it only needs two
small embedded controllers to work. Here is one:
http://www.semiconductorstore.com/cart/pc/viewPrd.asp?idproduct=12083

it's only $83, which is more than most motherboards, but hey ...

So, look for the COMA connector first. Last resort: usb debug port.

thanks

ron

[-- Attachment #2: Type: text/html, Size: 2354 bytes --]

Re: [9fans] 9pcf debugging

[erik quanstrom]

> I've seen there is rdbfs for debugging the 9pc kernel through a serial port...
>
> The computer the driver fails on doesn't have a serial port. I'd like
> to the debug the ethernet driver, so I don't have ethernet. :)
>
> Any ideas, over the common lot-of-print() ?
>
> Thanks in advance.

yes.  we have a small embedded kernel with no user space. since
it's hard to find serial these days, serial didn't seem like the best
solution.  so we export the segments, stack and memory as aoe
targets.  this won't help in the very early phases of boot, and
requires a working ethernet driver.  but our example is about
250 lines of code.  with aoedbfs (/n/sources/contrib/quanstro/src/aoedbfs)
acid or db can be used to debug the target with acid or db.

with the plan 9 kernel, i think the right solution would be to
put the aoe debugger in 9load.  this would require the plan 9
kernel leaving  9load in core (wasting 8 mb or so) and having
a mechanism for passing control back to 9load. this mechanism
could replace doublefault() and be inserted into debugbpt
and (on the 386) fault386.

- erik

p.s. there's also a program called aoesnap in
/n/sources/contrib/quanstro/src that will take a snapshot
of a process exporting debugging via aoe.  there may be some
assumptions that are unique to our situation and the code is
somewhat convoluted due to the fact that we typically snapshot
processes using gigabytes of memory, so the image needs to be
streamed out rather than processed in a serial manner like
snap.  it also contains compatability goo so it compiles on
linux.

Re: [9fans] 9pcf debugging

[Lluís Batlle]

Thank you all...
In fact I managed to get the ethernet driver working at the first code
change I tried today!
This means that by now I won't try to debug the kernel, but I really
like to have known of these kernel debugging techniques.

Btw, I had problems with the Rhine ethernet driver. I got much help
from #plan9... the driver wanted to align some structures to the cache
line size. The driver read the cls from the pci configuration space,
and my card had that register *bad* (0x04 instead of 0x08, for the x86
architecture I use).
As the driver allocates several Ds structures with a big malloc, and
dividing the allocated area to (p+=cls) parts, each for each Ds
structure, the driver required that cls >= sizeof Ds. That wasn't met
in my system, and the driver decided not to load.
I simply set manually the alignement to 32 bytes (0x08 dwords,
according to pci cls terminology), and it worked well for several
minutes, until I had to turn off the system.

Yesterday I tried aligning to 64 bytes, which I thought should work,
but it only worked for some tens of packets, and then the card
'hanged'. I cannot describe that hang better than the simple user
experience of no network packet receiving any answer, since some pings
worked and I could mount sources.

That's all. Poor helpful people in #9fans already know this. :)

2008/2/4, erik quanstrom <quanstro@coraid.com>:
> > I've seen there is rdbfs for debugging the 9pc kernel through a serial port...
> >
> > The computer the driver fails on doesn't have a serial port. I'd like
> > to the debug the ethernet driver, so I don't have ethernet. :)
> >
> > Any ideas, over the common lot-of-print() ?
> >
> > Thanks in advance.
>
> yes.  we have a small embedded kernel with no user space. since
> it's hard to find serial these days, serial didn't seem like the best
> solution.  so we export the segments, stack and memory as aoe
> targets.  this won't help in the very early phases of boot, and
> requires a working ethernet driver.  but our example is about
> 250 lines of code.  with aoedbfs (/n/sources/contrib/quanstro/src/aoedbfs)
> acid or db can be used to debug the target with acid or db.
>
> with the plan 9 kernel, i think the right solution would be to
> put the aoe debugger in 9load.  this would require the plan 9
> kernel leaving  9load in core (wasting 8 mb or so) and having
> a mechanism for passing control back to 9load. this mechanism
> could replace doublefault() and be inserted into debugbpt
> and (on the 386) fault386.
>
> - erik
>
> p.s. there's also a program called aoesnap in
> /n/sources/contrib/quanstro/src that will take a snapshot
> of a process exporting debugging via aoe.  there may be some
> assumptions that are unique to our situation and the code is
> somewhat convoluted due to the fact that we typically snapshot
> processes using gigabytes of memory, so the image needs to be
> streamed out rather than processed in a serial manner like
> snap.  it also contains compatability goo so it compiles on
> linux.
>

Re: [9fans] 9pcf debugging

[Martin Harriss]

ron minnich wrote:
...
>
> Ah, USB. 24+ chapters and not a single good idea :-)
> [ok, maybe one: powered cables are nice]
...

This should be a fortune (with or without the second line...

Martin

Re: [9fans] 9pcf debugging

[Kernel Panic]

Lluís Batlle wrote:

>Thank you all...
>In fact I managed to get the ethernet driver working at the first code
>change I tried today!
>This means that by now I won't try to debug the kernel, but I really
>like to have known of these kernel debugging techniques.
>
>Btw, I had problems with the Rhine ethernet driver. I got much help
>from #plan9... the driver wanted to align some structures to the cache
>line size. The driver read the cls from the pci configuration space,
>and my card had that register *bad* (0x04 instead of 0x08, for the x86
>architecture I use).
>As the driver allocates several Ds structures with a big malloc, and
>dividing the allocated area to (p+=cls) parts, each for each Ds
>structure, the driver required that cls >= sizeof Ds. That wasn't met
>in my system, and the driver decided not to load.
>I simply set manually the alignement to 32 bytes (0x08 dwords,
>according to pci cls terminology), and it worked well for several
>minutes, until I had to turn off the system.
>
>Yesterday I tried aligning to 64 bytes, which I thought should work,
>but it only worked for some tens of packets, and then the card
>'hanged'. I cannot describe that hang better than the simple user
>experience of no network packet receiving any answer, since some pings
>worked and I could mount sources.
>  
>
My VIA C7 based machine has the same problem:

0.18.0:	net  02.00.00 1106/3065  10 0:0000f001 256 1:fdffe000 256
	VIA Technology VT6102 Rhine II PCI Fast Ethernet Controller

so i could help you testing that thing :-)


cinap