From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=MAILING_LIST_MULTI, RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 6047 invoked from network); 28 Sep 2020 23:26:36 -0000 Received: from mother.openwall.net (195.42.179.200) by inbox.vuxu.org with ESMTPUTF8; 28 Sep 2020 23:26:36 -0000 Received: (qmail 32452 invoked by uid 550); 28 Sep 2020 23:26:29 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 32416 invoked from network); 28 Sep 2020 23:26:28 -0000 Date: Mon, 28 Sep 2020 19:26:15 -0400 From: Rich Felker To: musl@lists.openwall.com Message-ID: <20200928232614.GA17637@brightrain.aerifal.cx> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="ikeVEW9yuYc//A+q" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Subject: [musl] Pending patches for MT-fork stuff --ikeVEW9yuYc//A+q Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In investigating the MT-fork deadlock stuff and working on a patch that makes it attempt to trap rather than deadlocking, I found a problem with interaction of fork and aio. A fix for that, along with the patch to trap, and a minimal testcase for the aio bug, are attached. There's also a problematic interaction of abort with fork that was just found in glibc that also exists in musl, that I still need to fix. I'll follow up later with a proposed solution for that. Rich --ikeVEW9yuYc//A+q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0001-fix-fork-of-processes-with-active-async-io-contexts.patch" >From 34904d830a9fd1f6fc47218f38c111698303d2fe Mon Sep 17 00:00:00 2001 From: Rich Felker Date: Mon, 28 Sep 2020 18:38:27 -0400 Subject: [PATCH 1/3] fix fork of processes with active async io contexts previously, if a file descriptor had aio operations pending in the parent before fork, attempting to close it in the child would attempt to cancel a thread belonging to the parent. this could deadlock, fail, or crash the whole process of the cancellation signal handler was not yet installed in the parent. in addition, further use of aio from the child could malfunction or deadlock. POSIX specifies that async io operations are not inherited by the child on fork, so clear the entire aio fd map in the child, and take the aio map lock (with signals blocked) across the fork so that the lock is kept in a consistent state. --- src/aio/aio.c | 14 ++++++++++++++ src/internal/pthread_impl.h | 2 ++ src/process/fork.c | 3 +++ 3 files changed, 19 insertions(+) diff --git a/src/aio/aio.c b/src/aio/aio.c index 6d34fa86..f59679c3 100644 --- a/src/aio/aio.c +++ b/src/aio/aio.c @@ -392,6 +392,20 @@ int __aio_close(int fd) return fd; } +void __aio_atfork(int who) +{ + if (who<0) { + pthread_rwlock_rdlock(&maplock); + return; + } + if (who>0 && map) for (int a=0; a<(-1U/2+1)>>24; a++) + if (map[a]) for (int b=0; b<256; b++) + if (map[a][b]) for (int c=0; c<256; c++) + if (map[a][b][c]) for (int d=0; d<256; d++) + map[a][b][c][d] = 0; + pthread_rwlock_unlock(&maplock); +} + weak_alias(aio_cancel, aio_cancel64); weak_alias(aio_error, aio_error64); weak_alias(aio_fsync, aio_fsync64); diff --git a/src/internal/pthread_impl.h b/src/internal/pthread_impl.h index 4d709bbc..358ad1ce 100644 --- a/src/internal/pthread_impl.h +++ b/src/internal/pthread_impl.h @@ -162,6 +162,8 @@ extern hidden void *__pthread_tsd_main[]; extern hidden volatile int __aio_fut; extern hidden volatile int __eintr_valid_flag; +extern hidden void __aio_atfork(int); + hidden int __clone(int (*)(void *), void *, int, void *, ...); hidden int __set_thread_area(void *); hidden int __libc_sigaction(int, const struct sigaction *, struct sigaction *); diff --git a/src/process/fork.c b/src/process/fork.c index 7e984ff8..dbaa9402 100644 --- a/src/process/fork.c +++ b/src/process/fork.c @@ -10,6 +10,7 @@ static void dummy(int x) } weak_alias(dummy, __fork_handler); +weak_alias(dummy, __aio_atfork); pid_t fork(void) { @@ -17,6 +18,7 @@ pid_t fork(void) sigset_t set; __fork_handler(-1); __block_all_sigs(&set); + __aio_atfork(-1); #ifdef SYS_fork ret = __syscall(SYS_fork); #else @@ -32,6 +34,7 @@ pid_t fork(void) libc.threads_minus_1 = 0; if (libc.need_locks) libc.need_locks = -1; } + __aio_atfork(!ret); __restore_sigs(&set); __fork_handler(!ret); return __syscall_ret(ret); -- 2.21.0 --ikeVEW9yuYc//A+q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0002-add-hidden-declaration-for-internal-__aio_close-func.patch" >From 65c93ff68f93f83b471e52edd989c30d1c0f437f Mon Sep 17 00:00:00 2001 From: Rich Felker Date: Mon, 28 Sep 2020 18:47:13 -0400 Subject: [PATCH 2/3] add hidden declaration for internal __aio_close function --- src/internal/pthread_impl.h | 1 + 1 file changed, 1 insertion(+) diff --git a/src/internal/pthread_impl.h b/src/internal/pthread_impl.h index 358ad1ce..23ee566c 100644 --- a/src/internal/pthread_impl.h +++ b/src/internal/pthread_impl.h @@ -162,6 +162,7 @@ extern hidden void *__pthread_tsd_main[]; extern hidden volatile int __aio_fut; extern hidden volatile int __eintr_valid_flag; +extern hidden int __aio_close(int); extern hidden void __aio_atfork(int); hidden int __clone(int (*)(void *), void *, int, void *, ...); -- 2.21.0 --ikeVEW9yuYc//A+q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="0003-trap-don-t-deadlock-on-AS-unsafe-operations-after-mu.patch" >From 9179864fa24ecdabd2d2aadddf4d937cfb824090 Mon Sep 17 00:00:00 2001 From: Rich Felker Date: Mon, 28 Sep 2020 17:30:04 -0400 Subject: [PATCH 3/3] trap, don't deadlock, on AS-unsafe operations after multithreaded fork POSIX specifies the forked child of a multithreaded process to run in an async signal context where it cannot call AS-unsafe functions. the underlying motivation for this restriction is that internal resources in the parent may have been owned by other threads, which do not exist in the child, at the moment of fork, and may not be possible to access safely. prior to commit e01b5939b38aea5ecbe41670643199825874b26c, the child simply skipped locking due to its being single-threaded, resulting in silent use of inconsistent state, possibly leading to crashes, deadlocks, or other kinds of misbehavior. since that change, any attempt by the child to take a lock that was held in another thread of the parent reliably deadlocks. this prevents unsafe forward progress, but is an unexpected failure mode and difficult to log and handle in automated build and testing workflows. instead, track in the child that the fork was by a multithreaded parent, and if so, trap before waiting on a lock. this only works for internal locks serviced by LOCK/__lock, so it's possible that some things using other mechanisms will still deadlock, but this should catch most of the common problems and assist in finding and fixing erroneous application code. one subtlety that almost makes this change wrong is that we use threads internally for POSIX AIO and for SIGEV_THREAD timers. the existence of these internal threads cannot impose on what the application can do in the child. these subsystems don't use the affected locks themselves, but it's possible that a process that's only "multithreaded as an implementation detail" in the parent could fork, then legitimately create multiple threads in the child, leading to lock contention in the child that is not a result of UB by the application. in this case, however, pthread_create will have reset need_locks to a positive value, disarming the trap. --- src/process/fork.c | 2 +- src/thread/__lock.c | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/process/fork.c b/src/process/fork.c index dbaa9402..538a0bcb 100644 --- a/src/process/fork.c +++ b/src/process/fork.c @@ -32,7 +32,7 @@ pid_t fork(void) self->next = self->prev = self; __thread_list_lock = 0; libc.threads_minus_1 = 0; - if (libc.need_locks) libc.need_locks = -1; + if (libc.need_locks > 0) libc.need_locks = -2; } __aio_atfork(!ret); __restore_sigs(&set); diff --git a/src/thread/__lock.c b/src/thread/__lock.c index 60eece49..75412cc8 100644 --- a/src/thread/__lock.c +++ b/src/thread/__lock.c @@ -24,6 +24,7 @@ void __lock(volatile int *l) int current = a_cas(l, 0, INT_MIN + 1); if (need_locks < 0) libc.need_locks = 0; if (!current) return; + if (need_locks < -1) a_crash(); /* A first spin loop, for medium congestion. */ for (unsigned i = 0; i < 10; ++i) { if (current < 0) current -= INT_MIN + 1; -- 2.21.0 --ikeVEW9yuYc//A+q Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="aio_fork.c" #include #include #include int main() { char buf[256]; int fd = dup(0); struct aiocb cb = { .aio_fildes = fd, .aio_buf = buf, .aio_nbytes = sizeof buf }; aio_read(&cb); pid_t pid = fork(); if (!pid) { close(fd); _exit(0); } waitpid(pid, 0, 0); } --ikeVEW9yuYc//A+q--