Re: fork, set*id(synccall), cancellation -- nasty interaction

[Rich Felker <dalias@aerifal.cx>]

On Fri, Apr 26, 2013 at 01:27:22PM -0400, Rich Felker wrote:
> I've run across a nasty set of race conditions I'm trying to solve:
> 
> 1. __synccall, needed for multi-threaded set*id, needs to obtain a
> lock that prevents thread creation and other things. However, setuid
> and setgid are specified to be async-signal-safe. They will presently
> hang if called from a signal handler that interrupted pthread_create
> or several other functions.

I have a solution, but it's ugly and would decrease thread creation
performance by more than 10%. For static linked programs, the penalty
could be eliminated if __synccall is not used, but adding the logic to
do this would make the code much uglier.

The concept is fairly simple:

1. Fully protect the thread count by the "ptc" rwlock, so that if
another thread holds __inhibit_ptc(), the thread count can neither
increase nor decrease.

2. Prevent any calls to __synccall from interrupting code paths that
hold the "ptc" lock.

However, the requirements this translates into are:

1. pthread_create must block application signals unconditionally.
Right now it only does so in the special case of applying scheduling
changes to the new thread.

2. pthread_exit must perform the following acrobatics: First, block
application signals. Then, __acquire_ptc(). Then, block all signals,
then decrement the thread count and __release_ptc(). This adds both an
extra lock/unlock step and a second sigprocmask syscall to the exit
procedure. The two-step signal blocking is needed because, if all
signals were blocked at the time of the __acquire_ptc() call, it could
deadlock with another thread calling __synccall that had already
successfully performed __inhibit_ptc() and begun the broadcast.

3. The other user of __inhibit_ptc(), dlopen, would either need to
block signals for its duration, or the "ptc" rwlock could be replaced
by a two-way symmetric lock (allowing multiple 'readers' or multiple
'writers' but not both).

Basically, it's doable, but ugly. I'm still looking for better
solutions...

Rich