openmp/pthreads and fork...

openmp/pthreads and fork...

[Isaac Dunham]

Hello,
I've been packaging OpenBLAS for Alpine Linux, and an issue came to my
attention:
OpenBLAS (optionally) uses pthreads or OpenMP, and OpenMP uses pthreads.
OpenBLAS implements the BLAS (Basic Linear Algebra Subprograms) API/ABI,
like ATLAS, MKL, and so on.
Some programs that use BLAS will fork(); python is one of these.
With OpenBLAS, this had caused "random" segfaults due to use of threads
by the library both before and after the fork.
The OpenBLAS issue is here:
https://github.com/xianyi/OpenBLAS/issues/294

They worked around this using pthread_atfork() to cleanup before the fork().
I see some claims that calling any pthread functions in the child would be
UB, so I'm wondering about a few things:
-Is the last-mentioned claim correct?
-What is musl's behavior?
-How correct, and how likely to work with musl, is the fix for libgomp
mentioned here:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035
(and for that matter, the equivalent workaround referred to in OpenBLAS
issue 294)?
-Is there a safe (non-crashing) way to use/write libraries that might or
might not be built with threading--whether POSIX-specified or just
"working with musl" ?

Thanks,
Isaac Dunham

PS: OpenBLAS almost built out of the box; the only issue was use of
get_nprocs(), a GNU extension equivalent to sysconf(_SC_NPROCESSORS_ONLN).
When I submitted a patch, it was merged in about 2 hours.
I've been building with NO_AFFINITY=1 ... TARGET=ATOM.

Re: openmp/pthreads and fork...

[Szabolcs Nagy]

* Isaac Dunham <ibid.ag@gmail.com> [2014-08-04 22:56:51 -0700]:
> OpenBLAS (optionally) uses pthreads or OpenMP, and OpenMP uses pthreads.
> OpenBLAS implements the BLAS (Basic Linear Algebra Subprograms) API/ABI,
> like ATLAS, MKL, and so on.
> Some programs that use BLAS will fork(); python is one of these.
> With OpenBLAS, this had caused "random" segfaults due to use of threads
> by the library both before and after the fork.
> The OpenBLAS issue is here:
> https://github.com/xianyi/OpenBLAS/issues/294
> 
> They worked around this using pthread_atfork() to cleanup before the fork().
> I see some claims that calling any pthread functions in the child would be
> UB, so I'm wondering about a few things:
> -Is the last-mentioned claim correct?

the fix does make fork work with openmp in some cases:

if the application calls fork when no openmp code is
running then both the child and parent have the omp lib
in consistent state

if the application calls fork without synchronizing it
with openmp calls then the application is not allowed to
do much in the child anyway (it cannot call openmp code
so inconsistent state there is not an issue)
and the atfork handler is correct with some caveats:

- gomp_we_are_forked var should be volatile sig_atomic_t
in theory (but in practice their static int might work too)

- if fork happens in an async signal handler that interrupted
pthread_atfork, then the pthread_atfork global state may be
inconsistent so fork can crash in libc
(libc could probably guard against this but no libc does that
afaik, howver other libcs dont have an async-signal-safe fork
so this would be ub anyway)

> -What is musl's behavior?
> -How correct, and how likely to work with musl, is the fix for libgomp
> mentioned here:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035
> (and for that matter, the equivalent workaround referred to in OpenBLAS
> issue 294)?

the above workaround should work with musl (except when fork
is used form a sig handler)

with glibc there are other caveats:

- glibc dlclose removes the shared library, so if openmp was dlopened
and then dlclosed (may happen with python), then the next fork will
crash (because of the dangling atfork handler)
(the workaround is to use __register_atfork #ifdef __GLIBC__ which
takes care of the dlclose issue)

- pthread_atfork is only available in -lpthread, so a library that
is itself not multi-threaded but tries to work around fork issues
cannot use it without pulling in all the pthreads (__register_atfork
does not have this issue)

> -Is there a safe (non-crashing) way to use/write libraries that might or
> might not be built with threading--whether POSIX-specified or just
> "working with musl" ?

threading is not the issue, forking is (more specifically doing
library calls in the child after fork in a multi-threaded process)

similar issue came up recently in libressl
(see my notes starting at the "But.." section in
http://port70.net/~nsz/47_arc4random.html
)

my conclusion was that a library cannot reasonably do anything
to guard against unsafe fork usage in the application so it
shouldn't try to

pthread_atfork is broken, makes fork non-async-signal-safe and
cannot be used for the originally intended purposes

the application may not be aware of underlying threads in various
libraries, so fork should be avoided: use posix_spawn or
fork+exec (without intervening library calls), other fork usage
is just broken (unless the application has full control over
its library dependencies)

Re: openmp/pthreads and fork...

[Rich Felker]

On Mon, Aug 04, 2014 at 10:56:51PM -0700, Isaac Dunham wrote:
> Hello,
> I've been packaging OpenBLAS for Alpine Linux, and an issue came to my
> attention:
> OpenBLAS (optionally) uses pthreads or OpenMP, and OpenMP uses pthreads.
> OpenBLAS implements the BLAS (Basic Linear Algebra Subprograms) API/ABI,
> like ATLAS, MKL, and so on.
> Some programs that use BLAS will fork(); python is one of these.
> With OpenBLAS, this had caused "random" segfaults due to use of threads
> by the library both before and after the fork.
> The OpenBLAS issue is here:
> https://github.com/xianyi/OpenBLAS/issues/294
> 
> They worked around this using pthread_atfork() to cleanup before the fork().

This is unlikely to work. pthread_atfork is basically unable to do
what it was designed for, since, after fork, the new thread in the
child is not the owner of any mutexes that were obtained by the parent
before forking, and thus cannot unlock them. It also cannot destroy
and reinitialize them since they're locked.

It may be possible to work around this by using semaphores instead of
mutexes but I have not been able to get (and haven't really pushed
for) a definitive answer on whether this is valid (there's a question
whether using the forked copy of the semaphore is even valid; it's
certainly not, formally, if the original process was multi-threaded
since sem_wait is not AS-safe).

This issue is somewhat documented in the rationale for pthread_atfork:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html

But the way it's written, it's not at all clear that the result of the
analysis there is that using pthread_atfork just doesn't work. I
believe this is going to be changed in the next version of POSIX to
make it clear.

> I see some claims that calling any pthread functions in the child would be
> UB, so I'm wondering about a few things:
> -Is the last-mentioned claim correct?

Yes. See:
http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html

"A process shall be created with a single thread. If a multi-threaded
process calls fork(), the new process shall contain a replica of the
calling thread and its entire address space, possibly including the
states of mutexes and other resources. Consequently, to avoid errors,
the child process may only execute async-signal-safe operations until
such time as one of the exec functions is called. Fork handlers may be
established by means of the pthread_atfork() function in order to
maintain application invariants across fork() calls."

This is not stated clearly (what does "to avoid errors" mean? what
does "may only" mean?), but the important part is "the child process
may only execute async-signal-safe operations until such time as one
of the exec functions is called". This is generally taken as implying
that calls to any AS-unsafe function invoke UB.

> -What is musl's behavior?

musl makes no attept to synchronize malloc with fork, and doing so is
difficult if you want to maintain the current requirement that fork be
AS-safe (which is going to be removed in the next version of POSIX)
and somewhat bloated even if not difficult. A similar issue applies to
other code that uses locks internally -- stdio, time zone loading,
dlopen, etc. Grepping for __lock or LOCK would find most if not all
such uses.

Also, many synchronization primitives (e.g. all mutexes except
default/normal type) store their owner as a thread id, which is wrong
after fork. So attempting to use them after fork, if they were locked
before, is not going to work.

That's about all I'm aware of that breaks after fork, but there may be
more, and at present there's no intentional plan to avoid breaking
more.

> -How correct, and how likely to work with musl, is the fix for libgomp
> mentioned here:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035

Very unlikely since it's calling free which could run with malloc
locks permanently deadlocked.

> (and for that matter, the equivalent workaround referred to in OpenBLAS
> issue 294)?
> -Is there a safe (non-crashing) way to use/write libraries that might or
> might not be built with threading--whether POSIX-specified or just
> "working with musl" ?

As nsz said, using fork in this kind of way is just unsafe. It was
unsafe even without threads, but got a lot worse with threads. The
fundamental issues are that forking results in all resources that were
conceptually "owned" or "locked" by the parent having two owners after
the fork, and certain state (e.g. process id, but also lots of other
more subtle things) changes out from under the program in the child.

Rich

Re: openmp/pthreads and fork...

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2014-08-05 12:51:37 -0400]:
> On Mon, Aug 04, 2014 at 10:56:51PM -0700, Isaac Dunham wrote:
> > Some programs that use BLAS will fork(); python is one of these.
> > With OpenBLAS, this had caused "random" segfaults due to use of threads
> > by the library both before and after the fork.
> > The OpenBLAS issue is here:
> > https://github.com/xianyi/OpenBLAS/issues/294
> > 
> > They worked around this using pthread_atfork() to cleanup before the fork().
> 
> This is unlikely to work. pthread_atfork is basically unable to do
> what it was designed for, since, after fork, the new thread in the
> child is not the owner of any mutexes that were obtained by the parent
> before forking, and thus cannot unlock them. It also cannot destroy
> and reinitialize them since they're locked.

> > -How correct, and how likely to work with musl, is the fix for libgomp
> > mentioned here:
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035
> 
> Very unlikely since it's calling free which could run with malloc
> locks permanently deadlocked.
> 

the gomp patch says that it only fixes the case when fork is called
when no gomp threads are running (by default the threads are
kept around holding locks, they should be shut down before fork)

in that case it is enough to check a single global flag before
the new thread pool is initialized (and the flag can be set from
pthread_atfork child handler reasonably safely)

now i see that the openblas issue has all sorts of comments with
invalid solutions and in the end their fix is

  pthread_atfork(BLASFUNC(blas_thread_shutdown), NULL, NULL);

which should work unless fork was called from a sig handler

otherwise i dont see any call to free in AS-safe context