mailing list of musl libc
 help / color / mirror / code / Atom feed
From: Alexey Izbyshev <izbyshev@ispras.ru>
To: musl@lists.openwall.com
Cc: Ben Hillis <Ben.Hillis@microsoft.com>,
	Brian Perkins <Brian.Perkins@microsoft.com>,
	Peter Martincic <pmartincic@microsoft.com>,
	Pierre Boulay <Pierre.Boulay@microsoft.com>
Subject: Re: [musl] Potential deadlock when fork() is called while __aio_get_queue() is allocating memory
Date: Tue, 18 Oct 2022 20:07:34 +0300	[thread overview]
Message-ID: <f8ae3cd5d1b61b9d9b55b63ace4dbde2@ispras.ru> (raw)
In-Reply-To: <LV2PR21MB32536BE011E334AC9AE94041FA289@LV2PR21MB3253.namprd21.prod.outlook.com>

On 2022-10-18 19:51, Pierre Boulay wrote:

> Hello,
> 
> I'm working on the Windows Subsystem Linux, which uses MUSL and I'm 
> currently working on solving WSL complete freeze · Issue #8824 · 
> microsoft/WSL (github.com).
> 
> I believe that this freeze is caused by a bug causing a deadlock in 
> MUSL.
> 
> If we look at this sample program:
> 
> #include <unistd.h>
> #include <pthread.h>
> #include <aio.h>
> #include <fcntl.h>
> #include <time.h>
> #include <stdio.h>
> #include <errno.h>
> 
> void* thread2(void* unused) // Thread 2
> {
> unsigned char buf[256] = {0};
> struct aiocb aiocb = {};
> aiocb.aio_fildes = open("/proc/self/cmdline", O_RDONLY, S_IRUSR | 
> S_IWUSR);
> aiocb.aio_buf = buf;
> aiocb.aio_nbytes = sizeof(buf) - 1;
> aiocb.aio_lio_opcode = LIO_READ;
> 
> if (aio_read(&aiocb) < 0)
> {
> printf("aio_read failed, %i\n", errno);
> return NULL;
> }
> 
> int error = -1;
> while(error = aio_error(&aiocb) == EINPROGRESS)
> {
> printf("In progress...\n");
> sleep(1);
> }
> 
> if (error != 0)
> {
> printf("aio_error: %i\n", error);
> }
> 
> printf("aio result: %s\n", buf);
> return NULL;
> }
> 
> int main()
> {
> pthread_t thread;
> pthread_create(&thread, NULL, &thread2, NULL);
> 
> if (fork() > 0) // Thread 1
> {
> pthread_join(thread, NULL);
> printf("aio complete");
> }
> 
> }
> 
> Here we have two threads. The main thread is scheduling an asynchronous 
> io and the other is calling fork().
> 
> If we look at what the main thread is doing:
> 
> * After calling pthread_create, the fork() call goes here, in fork.c.
> * Before calling _Fork(), this thread will acquire the __malloc_lock 
> (through __malloc_atfork()),
> * Then _Fork() is called, which calls __aio_atfork(), which waits for 
> the maplock
> 
> In the meantime, the second thread will:
> 
> * Call aio_read(), which calls submit(), in aio.c
> * submit() then calls __aio_get_queue(), which acquires the maplock
> * At this point the map structure needs to be allocated so submit() 
> calls calloc()
> * calloc() calls malloc(), in malloc.c
> * If the allocation overflows, malloc() then calls wrlock(), which 
> waits for the __malloc_lock
> 
> So to summarize: Thread1 holds __malloc_lock and waits for maplock, and 
> thread2 holds maplock and waits for ___malloc_lock. We have a deadlock.
> 
> This is my understanding of the issue, but I have a (very) limited 
> knowledge of MUSL's codebase, so let me know if it's correct.
> 

Yes. Please see https://www.openwall.com/lists/musl/2022/10/06/2 and the 
following discussion.

Alexey

> If it is, I think this could be solved by  moving __aio_atfork() from 
> _Fork() to fork(), before __malloc_atfork() so the locks are acquired 
> in the correct order to prevent the deadlock.
> 
> I do see that _Fork() calls __block_all_sigs before calling 
> __aio_atfork() so I wonder if the order of these calls is important 
> though (let me know if that's the case).
> 
> If this solution makes sense to you, I'm happy to submit a contribution 
> with the fix.
> 
> Thank you,
> 
> --
> 
> Pierre Boulay

      reply	other threads:[~2022-10-18 17:07 UTC|newest]

Thread overview: 2+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-10-18 16:51 Pierre Boulay
2022-10-18 17:07 ` Alexey Izbyshev [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=f8ae3cd5d1b61b9d9b55b63ace4dbde2@ispras.ru \
    --to=izbyshev@ispras.ru \
    --cc=Ben.Hillis@microsoft.com \
    --cc=Brian.Perkins@microsoft.com \
    --cc=Pierre.Boulay@microsoft.com \
    --cc=musl@lists.openwall.com \
    --cc=pmartincic@microsoft.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/musl/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).