Re: Explaining cond var destroy [Re: [musl] C threads, v3.0]

[Rich Felker <dalias@libc.org>]

On Wed, Aug 06, 2014 at 10:43:15AM +0200, Jens Gustedt wrote:
> > If you think this is a bad idea, I'd be willing to hear alternate
> > ideas. I'm not really happy with the cond var implementation (if
> > nothing else, the sequence number thing is an ugly hack and not 100%
> > robust, I think) and at some point I'd like to redesign it.
> 
> As far as I can see the _c_destroy flag is used for no other purpose
> than this synchronization between the destroying thread and potential
> latecomers.

Yes, and I'm not even 100% sure the usage is correct and race-free. At
the very least the write should probably be a_store rather than simple
assignment.

> Technically the real problem is not pthread_cond_destroy (or
> cnd_destroy). This could just be a noop as it is for mutexes.

For mutexes, extreme care is taken that the implementation not use the
mutex object memory after the formal use (ownership) of the mutex
ends. This is why a mix of a waiter count and new-waiter flag on the
futex int itself is used (so that the waiter count does not need to be
read after the futex int is set to zero) and why the __vm_lock_* stuff
exists (to prevent munmap of the mutex while it's still in the pending
slot of the robust list; see glibc bug 14485).

> Using a
> condition after it is destroyed is UB in terms of the standards, but
> nothing hinders us to define a behavior for the specific
> implementation of musl.
> 
> It is the fact that the thread that calls destroy() might also
> deallocate the object directly after, and that the latecomers then
> crash because we removed their object under their feet. So this
> effectively introduces a deallocation barrier, which is nice, but
> clearly an extension.

It's not an extension. Such deallocation after destroy is explicitly
permitted and is actually the standard usage case, not the exception
(the natural time to destroy and free a cond var is right after you
put the predicate it's associated with into a permanently-true state
and signal all waiters; with cond vars in dynamically allocated
objects there may not even be a later time to do so).

The reason I went to all the trouble making a "deallocation barrier"
is because it's required.

> I am not sure about how to deal with this. The idea that destroy may
> be blocking or even be doing a context switch to the kernel came as a
> surprise to me. Maybe I would be happier if _c_destroy would be used
> as a usage count and destroy would just spinlock on that would be more
> straight.

A spinlock will deadlock if realtime priorities are used and the
destroying thread has higher priority than at least one of the
former-waiters that needs to return before destroy can proceed.

Fundamentally there's no reason context switch on destroy is odd.
POSIX (and C11) permits implementations where init/destroy for
synchronization objects requires some sort of kernel or kernel-like
allocation (e.g. in special memory). Light or zero-cost init/destroy
is a side effect of having a good underlying system, not a requirement
(though I would suspect many/most applications are not prepared for
systems where mutex init could fail, probably since good systems
guarantee it doesn't).

Rich