Re: Draft outline of thread-list design

[Alexey Izbyshev <izbyshev@ispras.ru>]

On 2019-02-15 01:32, Rich Felker wrote:
> On Fri, Feb 15, 2019 at 12:16:39AM +0300, Alexey Izbyshev wrote:
>> On 2019-02-12 21:26, Rich Felker wrote:
>> >pthread_join:
>> >
>> >A joiner can no longer see the exit of the individual kernel thread
>> >via the exit futex (detach_state), so after seeing it in an exiting
>> >state, it must instead use the thread list to confirm completion of
>> >exit. The obvious way to do this is by taking a lock on the list and
>> >immediately releasing it, but the actual taking of the lock can be
>> >elided by simply doing a futex wait on the lock owner being equal to
>> >the tid (or an exit sequence number if we prefer that) of the exiting
>> >thread. In the case of tid reuse collisions, at worse this reverts to
>> >the cost of waiting for the lock to be released.
>> >
>> Since the kernel wakes only a single thread waiting on ctid address,
>> wouldn't the joiner still need to do a futex wake to unblock other
>> potential waiters even if it doesn't actually take the lock by
>> changing *ctid?
> 
> I'm not sure. If it's just a single wake rather than a broadcast then
> yes, but only if it waited. If it observed the lock word != to the
> exiting thread tid without performing a futex wait then it doesn't
> have to do a futex wake.
> 
Yes, it's a single wake: 
<http://man7.org/linux/man-pages/man2/set_tid_address.2.html>, 
<https://elixir.bootlin.com/linux/v4.20.8/source/kernel/fork.c#L1292>.
> 
>> In general, to my limited expertise, the design looks simple and
>> clean. I'm not sure whether it's worth optimizing to reduce
>> serialization pressure on pthread_create()/pthread_exit() because
>> creating a large amount of short-lived threads doesn't look like a
>> good idea anyway.
> 
> Yes. One thing I did notice is that the window where pthread_create
> has to hold a lock to prevent new dlopen from happening is a lot
> larger than the window where the thread list needs to be locked, and
> contains mmap/mprotect. I think we should add a new "DTLS lock" here
> that's held for the whole time, with a protocol that if you need both
> the DTLS lock and the thread list lock, you take them in that order
> (dlopen would also need them both). This reduces the thread list lock
> window to just the __clone call and list update.
> 
Looks good.

Alexey