[REVIEW] 16497 make gets lost in make clobber if DMAKE_ADJUST_MAX_JOBS=M2

[REVIEW] 16497 make gets lost in make clobber if DMAKE_ADJUST_MAX_JOBS=M2

[Bill Sommerfeld]

Issue: https://www.illumos.org/issues/16497
CR: https://code.illumos.org/c/illumos-gate/+/3434
Diff: 
https://code.illumos.org/~diff/2d22cc063fd1fe282e12c8b9fe5f0b63c0eceeed

I've made some changes to our make(1) that improve its ability to
limit the load it puts on the build system.

An inherent problem in parallel recursive make is that the offered
load on the build system can grow roughly exponentially in the depth
of the source tree if there is no governor limiting the rate at which
new tasks are created.

Other versions of make (notably GNU make and some versions of BSD
pmake[1]) use IPC mechanisms to put a strict cap on the number of
parallel jobs.

The version of make in our tree will, by default, limit jobs with a
periodic check of the system's load average.  This works ok, but load
average is a lagging indicator -- it may take a while for it to kick
in, leading to uneven offered load.

It turns out that there is also an undocumented setting
("DMAKE_ADJUST_MAX_JOBS=M2") which, like GNU Make, attempts to put a
strict cap on the number of jobs within a tree of make processes; it
uses a counting semaphore created by the topmost make process.  But
the code which does this is buggy in a couple different ways[2], and
it used a broken algorithm for token accounting.

I believe I've fixed these bugs and have been able to run a half dozen
or so nightly builds on my machine with various versions of these
changes.  If used in in conjunction with CW_SHADOW_SERIAL it keeps the
system's load average during a nightly from going much above
$DMAKE_MAX_JOBS and may shave a few percent off the build duration.

I'd appreciate a look at these changes and would like to get it out
into more hands for others to experiment with.

Thanks in advance for your review!

Footnotes:

[1] Long ago I had a hand in adding this to the NetBSD version of make.

[2] If you put DMAKE_ADJUST_MAX_JOBS=M2 in the environment file of
nightly, you will not get a successful build.  The build goes off the
rails early on, somewhere in the "make clobber" pass.  Somehow a
recursive make invocation ends up being corrupted, with the result
that the "make clobber" pass ends up running a "make all" in part of
the tree before the tools have been built.  I haven't dug into exactly
how or why this trainwreck occurs, but when I fixed the M2 governor to
mimic the default governor's interaction with the rest of make, the
problem went away.