It's quite odd.  The listener doesn't do anything while waiting for a call,
i.e., sshserve doesn't get started until the call is made.  No matter how
badly you misconfigured it, it won't do anything until you connect to
the port.  It sounds more like your interrupts stop when you have something
to run.

What hardware are you doing this on?  What does /dev/sysstat show
about the interrupt load?