[musl] Getting rid of vmlock

[Rich Felker <dalias@libc.org>]

The vmlock mechanism is a horrible hack to work around Linux's robust
mutex system being broken; it inherently has a race of the following
form:

	thread A		thread B
1.	put M in pending
2.	unlock M
3.				lock M
4.				unlock M
5.				destroy M
6.				unmap M's memory
7.				mmap something new at same address
8.	remove M from pending

whereby, if the process is async terminated (e.g. by SIGKILL) right
before the line 8 (removing M from pending), the kernel will clobber
unrelated memory. Normally this is harmless since the process is
dying, but it's possible this unrelated memory is a new MAP_SHARED
PROT_WRITE file mapping, and now you have file corruption.

musl's vmlock mechanism avoids this by taking a sort of lock (just a
usage counter) while a robust list pending slot is in use, and
disallowing munmap or mmap/mremap with MAP_FIXED until the counter
reaches zero. This has the unfortunate consequence of making the mmap
family non-AS-safe. Thus I'd really like to get rid of it.

Right now we're preventing the problem by forcing the munmap operation
(line 6) to wait for the delisting from pending (ling 8). But we could
instead just delay the lock (line 3) of a process-shared robust mutex
until all pending operations in the current process finish. Since the
window (between line 2 and 8) only exists until pthread_mutex_unlock
returns in thread A, there is no way for thread B to observe M being
ready to destroy/unmap/whatever without a lock or trylock operation on
M. So this seems to work.

Both before and after such a change, however, we have a nasty problem.
If thread A is interrupted by a signal handler, or has low priority
(think PI where the unlock drops priority), between 2 and 8, it's
possible for the pending slot removal to be delayed arbitrarily. This
is really bad.

I think the problem is solvable by not just keeping a count of threads
with pending robust list slots, but making them register in a list (or
simply using the existing global thread list). Then when thread B
takes the lock (line 3), rather than waiting on the pending count to
reach zero, if it sees a nonzero count, it can walk the list and
*perform the removal itself* of M from any pending slot it finds M in.

The vmlock is also used (rather gratuitously, in the same way) for
process-shared barriers. These can be fixed to perform __vm_wait
before returning, rather than imposing that it happen later at unmap
time, and then the vmlock can be replaced with a "threads exiting
pshared barrier" lock that's functionally equivalent to the old
vmlock.


Rich