Re: openmp/pthreads and fork...

[Rich Felker <dalias@libc.org>]

On Mon, Aug 04, 2014 at 10:56:51PM -0700, Isaac Dunham wrote:
> Hello,
> I've been packaging OpenBLAS for Alpine Linux, and an issue came to my
> attention:
> OpenBLAS (optionally) uses pthreads or OpenMP, and OpenMP uses pthreads.
> OpenBLAS implements the BLAS (Basic Linear Algebra Subprograms) API/ABI,
> like ATLAS, MKL, and so on.
> Some programs that use BLAS will fork(); python is one of these.
> With OpenBLAS, this had caused "random" segfaults due to use of threads
> by the library both before and after the fork.
> The OpenBLAS issue is here:
> https://github.com/xianyi/OpenBLAS/issues/294
> 
> They worked around this using pthread_atfork() to cleanup before the fork().

This is unlikely to work. pthread_atfork is basically unable to do
what it was designed for, since, after fork, the new thread in the
child is not the owner of any mutexes that were obtained by the parent
before forking, and thus cannot unlock them. It also cannot destroy
and reinitialize them since they're locked.

It may be possible to work around this by using semaphores instead of
mutexes but I have not been able to get (and haven't really pushed
for) a definitive answer on whether this is valid (there's a question
whether using the forked copy of the semaphore is even valid; it's
certainly not, formally, if the original process was multi-threaded
since sem_wait is not AS-safe).

This issue is somewhat documented in the rationale for pthread_atfork:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html

But the way it's written, it's not at all clear that the result of the
analysis there is that using pthread_atfork just doesn't work. I
believe this is going to be changed in the next version of POSIX to
make it clear.

> I see some claims that calling any pthread functions in the child would be
> UB, so I'm wondering about a few things:
> -Is the last-mentioned claim correct?

Yes. See:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html

"A process shall be created with a single thread. If a multi-threaded
process calls fork(), the new process shall contain a replica of the
calling thread and its entire address space, possibly including the
states of mutexes and other resources. Consequently, to avoid errors,
the child process may only execute async-signal-safe operations until
such time as one of the exec functions is called. Fork handlers may be
established by means of the pthread_atfork() function in order to
maintain application invariants across fork() calls."

This is not stated clearly (what does "to avoid errors" mean? what
does "may only" mean?), but the important part is "the child process
may only execute async-signal-safe operations until such time as one
of the exec functions is called". This is generally taken as implying
that calls to any AS-unsafe function invoke UB.

> -What is musl's behavior?

musl makes no attept to synchronize malloc with fork, and doing so is
difficult if you want to maintain the current requirement that fork be
AS-safe (which is going to be removed in the next version of POSIX)
and somewhat bloated even if not difficult. A similar issue applies to
other code that uses locks internally -- stdio, time zone loading,
dlopen, etc. Grepping for __lock or LOCK would find most if not all
such uses.

Also, many synchronization primitives (e.g. all mutexes except
default/normal type) store their owner as a thread id, which is wrong
after fork. So attempting to use them after fork, if they were locked
before, is not going to work.

That's about all I'm aware of that breaks after fork, but there may be
more, and at present there's no intentional plan to avoid breaking
more.

> -How correct, and how likely to work with musl, is the fix for libgomp
> mentioned here:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035

Very unlikely since it's calling free which could run with malloc
locks permanently deadlocked.

> (and for that matter, the equivalent workaround referred to in OpenBLAS
> issue 294)?
> -Is there a safe (non-crashing) way to use/write libraries that might or
> might not be built with threading--whether POSIX-specified or just
> "working with musl" ?

As nsz said, using fork in this kind of way is just unsafe. It was
unsafe even without threads, but got a lot worse with threads. The
fundamental issues are that forking results in all resources that were
conceptually "owned" or "locked" by the parent having two owners after
the fork, and certain state (e.g. process id, but also lots of other
more subtle things) changes out from under the program in the child.

Rich