What does this code do?

What does this code do?

[Jens Eichendorff]

While skimming some parts of the musl libc I have come across the
following weird line in `src/thread/__wait.c':

  if (priv) priv = 128; priv=0;

whereby `priv' is then bitwise OR-ed to the FUTEX_WAIT operation
constant to a futex syscall.  Does this code exist to avoid
optimizations, do optimizations, or just add waiting time?  What does it
do and what is the `priv' parameter of __wait() for?  Excuse me if this
is a really stupid question, I'm not very familiar with somewhat
low-level code.

Looking through the git logs this is part of the initial import commit,
so I could not find any comments to why this was done.

MfG
Jens

Re: What does this code do?

[Rich Felker]

On Sat, Feb 22, 2014 at 03:35:43PM +0100, Jens Eichendorff wrote:
> While skimming some parts of the musl libc I have come across the
> following weird line in `src/thread/__wait.c':
> 
>   if (priv) priv = 128; priv=0;
> 
> whereby `priv' is then bitwise OR-ed to the FUTEX_WAIT operation
> constant to a futex syscall.  Does this code exist to avoid
> optimizations, do optimizations, or just add waiting time?  What does it
> do and what is the `priv' parameter of __wait() for?  Excuse me if this
> is a really stupid question, I'm not very familiar with somewhat
> low-level code.
> 
> Looking through the git logs this is part of the initial import commit,
> so I could not find any comments to why this was done.

It's not a stupid question at all. This is simply a silly way of
disabling support for private-futex mode, which is not presently
working. Private futex mode is a feature added to the kernel somewhere
during the mid 2.6 series (if I recall right) that only works for
futexes within a single process, not shared between processes, by
using the virtual address directly as the futex key rather than keying
the futex by the underlying backing for the mapping. It allows for a
much faster path in kernel-space (avoids taking a global lock and
possibly sleeping) and possibly also eliminates some ugly failure
cases (which in principle should not exist, but the kernel is broken
like that...).

Unfortunately, since we want to support both old and new kernels
(where old kernels lack it), we need a way to fallback to not using
private futex mode if it's not supported. Also, the synchronization
object needs to accurately reflect whether the application requested
it be process-shared, so that we can avoid using private futex mode
when it's not valid to do so. Making the fallback reasonably efficient
is not easy; we want to cache the result, but storing it globally
would incur synchronization cost and GOT access cost on each futex
operation (potentially expensive in bloat as well as time) while
storing it in TLS would incur thread-pointer access time (very slow on
some archs) for each operation and have the risk of threads
mismatching each other's choices due to spurious failures by the
kernel (not sure if these exist or not) that could lead to deadlocks.
So adding support, while it's been an agenda item for a long time,
requires nontrivial work/research on how to do it right (note: I'm not
convinced glibc does it right).

Hope this answers your question!

Rich

Re: What does this code do?

[Rich Felker]

On Sat, Feb 22, 2014 at 09:59:07AM -0500, Rich Felker wrote:
> Unfortunately, since we want to support both old and new kernels
> (where old kernels lack it), we need a way to fallback to not using
> private futex mode if it's not supported. Also, the synchronization
> object needs to accurately reflect whether the application requested
> it be process-shared, so that we can avoid using private futex mode
> when it's not valid to do so. Making the fallback reasonably efficient
> is not easy; we want to cache the result, but storing it globally
> would incur synchronization cost and GOT access cost on each futex
> operation (potentially expensive in bloat as well as time) while
> storing it in TLS would incur thread-pointer access time (very slow on
> some archs) for each operation and have the risk of threads
> mismatching each other's choices due to spurious failures by the
> kernel (not sure if these exist or not) that could lead to deadlocks.
> So adding support, while it's been an agenda item for a long time,
> requires nontrivial work/research on how to do it right (note: I'm not
> convinced glibc does it right).

I may have a solution for this that avoids any costly synchronization
or TLS access: check the availability of private futexes during the
first call to pthread_create. No synchronization is required at this
point (since no other threads exist yet) and no operation can care
whether private futexes are supported prior to pthread_create (since
there are no threads that can contend for non-process-shared
synchronization objects). We can store the private futex flag in the
"libc" structure too so it's accessible via PC-relative/GOT-relative
addressing rather than needing a GOT lookup.

Unless anyone sees serious flaws in this approach (note: the extra
futex syscall to test for private futex availability during the first
call to pthread_create has nonzero cost, but it's dwarfed by the cost
of clone and other operations, so IMO that's not an issue) I'll plan
to go with this approach to adding private futex support. I don't want
to risk breakage now though so I'll do it after 1.0, possibly as its
own change in 1.0.1 or so, and otherwise as part of the changes to
make the thread pointer always-valid in the 1.1.x series.

Rich