runsv and EAGAIN

[Charlie Brady <charlieb-supervision@budge.apana.org.au>]

I'm just putting together a new sytem using CentOS 5.2 (i686 - 
2.6.18-92.el5PAE), and have runsv chewing up lots of CPU:

...
  1202 root      25   0   108   28   20 R 10.2  0.0   3:13.97 runsv
  1300 root      25   0   112   36   20 R 10.2  0.0   3:17.01 runsv
  1306 root      25   0   108   28   20 R 10.2  0.0   3:18.52 runsv
  1322 root      25   0   112   36   20 R 10.2  0.0   3:16.97 runsv
  1203 root      25   0   108   32   20 R  9.9  0.0   3:19.21 runsv
  1228 root      25   0   112   36   20 R  9.9  0.0   3:21.39 runsv
  1271 root      25   0   112   32   20 R  9.9  0.0   3:24.37 runsv
  1200 root      25   0   112   32   20 R  9.6  0.0   3:23.17 runsv
...

Here's a sample strace:

rt_sigprocmask(SIG_UNBLOCK, [TERM], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
poll([{fd=3, events=POLLIN}, {fd=9, events=POLLIN, revents=POLLIN}, 
{fd=11, events=POLLIN}], 3, 1000020) = 1
rt_sigprocmask(SIG_BLOCK, [TERM], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
read(3, 0xbfa256c3, 1)                  = -1 EAGAIN (Resource temporarily 
unavailable)
waitpid(-1, 0xbfa256bc, WNOHANG)        = 0
read(9, "", 1)                          = 0
read(11, 0xbfa256c3, 1)                 = -1 EAGAIN (Resource temporarily 
unavailable)
gettimeofday({1215792640, 591768}, NULL) = 0
rt_sigprocmask(SIG_UNBLOCK, [TERM], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [CHLD], NULL, 8) = 0
poll([{fd=3, events=POLLIN}, {fd=9, events=POLLIN, revents=POLLIN}, 
{fd=11, events=POLLIN}], 3, 1000020) = 1
rt_sigprocmask(SIG_BLOCK, [TERM], NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], NULL, 8) = 0
read(3, 0xbfa256c3, 1)                  = -1 EAGAIN (Resource temporarily 
unavailable)
waitpid(-1, 0xbfa256bc, WNOHANG)        = 0
read(9, "", 1)                          = 0
read(11, 0xbfa256c3, 1)                 = -1 EAGAIN (Resource temporarily 
unavailable)
gettimeofday({1215792640, 593313}, NULL) = 0
...

I think this might be a case of a kernel being "this dumb" - iopause.c:

...
  	/* XXX: some kernels apparently need x[0] even if len is 0 */
  	/* XXX: how to handle EAGAIN? are kernels really this dumb? */
  	/* XXX: how to handle EINVAL? when exactly can this happen? */
...


...
ERRORS
        The poll() function shall fail if:

        EAGAIN The allocation of internal data structures failed but a 
subsequent request may succeed.

...

The system isn't obviously out of memory:

[root@amc-upgrade:]# free
              total       used       free     shared    buffers     cached
Mem:        255608     213396      42212          0      29240     109132
-/+ buffers/cache:      75024     180584
Swap:       265064          0     265064
~
[root@amc-upgrade:]#

[root@amc-upgrade:]# cat /proc/meminfo
MemTotal:       255608 kB
MemFree:         47580 kB
Buffers:         29276 kB
Cached:         109176 kB
SwapCached:          0 kB
Active:         112132 kB
Inactive:        75084 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       255608 kB
LowFree:         47580 kB
SwapTotal:      265064 kB
SwapFree:       265064 kB
Dirty:            2176 kB
Writeback:           0 kB
AnonPages:       48780 kB
Mapped:           9236 kB
Slab:            15600 kB
PageTables:       2372 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:    392868 kB
Committed_AS:   206484 kB
VmallocTotal:   772088 kB
VmallocUsed:      2284 kB
VmallocChunk:   769652 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB
~
[root@amc-upgrade:]#