[9fans] installs which hang

[9fans] installs which hang

[erik quanstrom]

i don't think this has been mentioned in a while, so i wanted
to quickly jot down a few tricks for looking at a "hung" machine.

there are three main things that can hang things up on initial boot
that aren't related to the kernel misbehaving
- unclean shutdown forcing a fs check.  this can take 15-20 minutes on a
big drive, or slow vm disk.
- dhcp (should be limited to 5-10 minutes on sources, and ~30s on atom)
- timesync.  i saw this issue one in 2008, so i don't remember much about it.
- interrupts incorrectly mapped.  (even acpi can get it wrong.)

if you can hit enter on the console, and have it recognized, you can
have some confidence that interrupts are working.

if you can type ^T^Tp and get a process listing, that's even better.
you may (especially if you have a serial console) be able to figure out who
is misbehaving.

if you can type ^T^Tq and get the scheduler dump that might tell you
if you have a lot of runnable processes.

if you can type ^T^Ti and get the interrupt dump, that might tell you
if a some hardware isn't interrupting.

if your machine isn't making it this far, there isn't too much that can be
easily done, unless you can pxe boot.  i usually put prints in the boot to
see where things are going wrong.  i had the pleasure of doing that yesterday
putting new locks in the pae kernel.  (i really need to use charles' GS:
extern register trick to avoid this MACHP nonsense.)

- erik

Re: [9fans] installs which hang

[Ramakrishnan Muthukrishnan]

On Sat, Jun 7, 2014 at 9:15 PM, erik quanstrom <quanstro@quanstro.net> wrote:
> i don't think this has been mentioned in a while, so i wanted
> to quickly jot down a few tricks for looking at a "hung" machine.
>
> there are three main things that can hang things up on initial boot
> that aren't related to the kernel misbehaving
> - unclean shutdown forcing a fs check.  this can take 15-20 minutes on a
> big drive, or slow vm disk.
> - dhcp (should be limited to 5-10 minutes on sources, and ~30s on atom)
> - timesync.  i saw this issue one in 2008, so i don't remember much about it.
> - interrupts incorrectly mapped.  (even acpi can get it wrong.)
>
> if you can hit enter on the console, and have it recognized, you can
> have some confidence that interrupts are working.
>
> if you can type ^T^Tp and get a process listing, that's even better.
> you may (especially if you have a serial console) be able to figure out who
> is misbehaving.
>
> if you can type ^T^Tq and get the scheduler dump that might tell you
> if you have a lot of runnable processes.
>
> if you can type ^T^Ti and get the interrupt dump, that might tell you
> if a some hardware isn't interrupting.
>
> if your machine isn't making it this far, there isn't too much that can be
> easily done, unless you can pxe boot.  i usually put prints in the boot to
> see where things are going wrong.  i had the pleasure of doing that yesterday
> putting new locks in the pae kernel.  (i really need to use charles' GS:
> extern register trick to avoid this MACHP nonsense.)

Thank you. This is very useful.

With the 9atom install on virtualbox which hangs at init: starting
/bin/rc, I can see the process listing with ^T^Tp. The scheduler
listing just show "nrdy 0". ^T^Ti does not show anything. Keyboard is
active.

--
  Ramakrishnan

Re: [9fans] installs which hang

[Steve Simon]

> - timesync.  i saw this issue one in 2008, so i don't remember much about it.

I think this was a bug in cron. When the time lept forward as timesync corrected
the time at boot cron would try to run all the intervening events and hang the machine.

cron now ignores time changes if they are big.

Having said this my memory is a little hazy too...

-Steve

Re: [9fans] installs which hang

[erik quanstrom]

On Sat Jun  7 19:22:41 EDT 2014, steve@quintile.net wrote:
> > - timesync.  i saw this issue one in 2008, so i don't remember much about it.
>
> I think this was a bug in cron. When the time lept forward as timesync corrected
> the time at boot cron would try to run all the intervening events and hang the machine.
>
> cron now ignores time changes if they are big.
>
> Having said this my memory is a little hazy too...

it can also lock up the machine updating "too fast".  without help
from cron.  i just don't remember the details of why this resulted
in a hang.

- erik