From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HTML_MESSAGE,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2 autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 13538 invoked from network); 18 Oct 2022 16:57:28 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 18 Oct 2022 16:57:28 -0000 Received: (qmail 32172 invoked by uid 550); 18 Oct 2022 16:57:21 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 28537 invoked from network); 18 Oct 2022 16:52:10 -0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jFucJ2Ko9fOWDqryUgMIdjItPVuQLom3bIoCaF6c7YE97hKgfQR/aptm6hKnFMT7dVsAWt4UJoFFDtFL3F6uYTJh2xFtsKLlOE3J8b29JcHmFK6t4pFOfq1qNz7DtG0WDSp9rJ7hK2oedJyo22LJow4AK2T0yIKHfIeFirvDgtK+1yWF7l74s23OOYQQIkv0hW3ZWY6pfPdEAK2vMLR6YSyQrKnpkrDhUQXNaExXYWJhfOVunV9bk8FUAtVjq4K7Enj0DowG2GYRuLBOmUE9Cu0259ieGPmAq2dcQDCIXNe0DUeI7QvX/FIYvriySB4HAVswbzFbYqHpAwXnz8sSNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xq5CUjGd02X6mesYUugl8M3U7aELrp4E52r7tisAPzo=; b=B9pEgW2RtkqVaVKjEQfGBmleMbR+fZfTlJBkVhcvATjoaUBULRQbIUroqzpulSKif3Zxw3hvNhWIOxo1aTMzpippk/gUbg1uofQU+QmA+yJ4qpjSSGExw0QamfyG28sIY01h7nq+o32LkzGCNtB3WmcrpxhGOxH8hXEjKVzRpH9yi7uhnAauWt2xI+9XdrlmCLnOo70jIy97QVbGmuWwjIuH/HZ9BgncKfXD/VgL8nuTXKL8Gp/6PHGBzMA88pXDbavPawRgy3VXVHvLX3BVGlx4irLLPlTg4FBlOa9cX2NBmKQMqFRI8E0GqwgHG9j28xH7ctQmEF2Ly/m9i5jRTg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xq5CUjGd02X6mesYUugl8M3U7aELrp4E52r7tisAPzo=; b=RIqZevmY+wUeY9Qj1/QvVcGUcrB6ecm+zobFMpp/Ind/u1nZTGckkDA2aM/M9UpyWYv0DNR5ytOBZc4EZjphSglkepsK3agsAdEC2s+Epik0KpZCAanXWteCDHK7ood9Nunr0Jj41Gv/Jw1YQd4Sp5NVfcCKyvukp1BS9zYSWXY= From: Pierre Boulay To: "musl@lists.openwall.com" CC: Ben Hillis , Brian Perkins , Peter Martincic Thread-Topic: Potential deadlock when fork() is called while __aio_get_queue() is allocating memory Thread-Index: AdjjCY/gvsz0JLtMQEe7/UQ17y5Veg== Date: Tue, 18 Oct 2022 16:51:56 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=18c6c24b-fe6a-4cde-9d1d-1c406d3913cf;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2022-10-18T15:49:52Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=microsoft.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: LV2PR21MB3253:EE_|SJ0PR21MB1999:EE_ x-ms-office365-filtering-correlation-id: f5b32ea9-3156-42e4-345f-08dab1291108 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: MhE6tg0z6Irl5s726KPb+DPbBQaF910eboNZSORMzHiXBYuv6pt9jdiBBb/oGM98ma4srzYyvd7CMnOmcRSHnxYNhhJNqNxnWpJFyT0t3agPQSO7oySvRYVSgDqW49IZMTMIHhZrANZNpdI65WvGj+/KyQNUpqe6Zyp1wCbJQoMc6mQh8zDakqKozrcCGEX6Nqf1REZjMnuKvSvqjl4ihdQByQrpmW2X6dE7tLjB6ASncDd1Bu7t/mpdjmYJ040le6CkVaPdbC6GosBLdMwX9XGtQHfClIbY6FuNwP+dZ0mxn/oo0RCqQCgEyyKEH9HSzI/1rW3a71DQS1gNTcCTxcwKyDqCPXy1PVTzzvbK19JOc1IeJAYWOioy8Io3Ik3CxVcwhSKBS3rK1H9ybuc5xxJYFLIROH2UOrKW+DvMkyIlnt0Nm+lnC3ktOMxUGGlTCbijKn97CdSfr5IyqxzUEucVtRnoKKMSzfDsyphs5UFZX+4zJcr5dJaIsmH2d5jwjN/6kvXJRqUUHJMIoWNimAYocHRLLcLodyJ/LI874iP0pTPpFeCKTNJrahq5teUQ/PoTWE3v7CnNhqcXrSmPhOq4BqS5dfyNUabu1K9h51sM+nZtlhxU1rm6Fy+OA/cmJTxgWBKxBGSN3OBdK/k+Cle+TqT5x92igOWK+HPD/ivzqDUWf07oALFGU6n0qalKu2IJKhSbxz3Lnt2kbRKkYQNDoeu32reNk+5Vcg2piy5dOh2RskDR7+PLq9ntqTRRG8wATVYmvjI8sz5pNnn7sAC2N3kGti/Us2KbV2kPmPuUaK02qwtmyFGZxA5/6eTbEGwbjDJp7k+xPsMCaYELCg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR21MB3253.namprd21.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(376002)(366004)(136003)(39860400002)(346002)(396003)(451199015)(71200400001)(478600001)(33656002)(83380400001)(10290500003)(54906003)(6916009)(316002)(4326008)(64756008)(66556008)(66946007)(7696005)(9686003)(66446008)(8936002)(5660300002)(41300700001)(9326002)(76116006)(6506007)(186003)(55016003)(8990500004)(166002)(2906002)(66476007)(82950400001)(82960400001)(8676002)(107886003)(86362001)(52536014)(38100700002)(38070700005)(122000001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LepAsvcukVlWqkwlSA2dboGpzIHY9sYjliqwnnlHRwYmiHoLQrHFke7TWtao?= =?us-ascii?Q?ErpCsvLM262quSef5Qudvs3mKtdF461OMwol6j0g8PYGvIOp+/ShtRVpQDYF?= =?us-ascii?Q?Luo2wzMRjdCbbZEw9iwr71DvkeC6TvQqQX/Pz0T94XSvsd4D8mN7wrXgmbTF?= =?us-ascii?Q?7eO1Cge+6Hpw7HxedyGZmj/XAePnSSqV39yfIFT+lW2lpHDrWjQpsnj+oB0D?= =?us-ascii?Q?tJQvPcnsun8vyU7pDzFRuzYqR+0tCMHLUmLSBeWgwyeC/stu/XiDru537vC9?= =?us-ascii?Q?xtRKodthJidyU/UNM0D1i74t5Ka5kCSe5JGx6870CviijKf9HdIgjKB6PWgb?= =?us-ascii?Q?cwo6W92FZtKlZSmrqYD3HF0GHXwLnAIzuhkSDoRSQtaIISVcrrDZUW9bKezx?= =?us-ascii?Q?YlNIX2OsZk/InOyDy0taFjfAcvCG45J+9DzezYxY/1M1HdIqwr+x/H/8HSeF?= =?us-ascii?Q?rvtqx2QqeP9UTWUEBiz0Vuah5z4WocYiC8l/IgHgSjnxxdHYIFFtc5wSVxmA?= =?us-ascii?Q?yZsqLxqpJJb164LbQAor6XdTGNIUPyuhmG7Uf+G/pX7didABzEGpbojY38sQ?= =?us-ascii?Q?qNeLnH0dF3ntaWDiVNLkpeRhIG8fU1UgCmH9SWVKdAZHX8oPI8urSLUZz1LZ?= =?us-ascii?Q?daeJ44TDJk/V/5s6h7veArS97WZMoXFByAofkyNqdIL7MjdPDuKCTVrVj+Oj?= =?us-ascii?Q?3nq+OrhX6A2T+fpCWNDnr4tS71c98EzVuQHeL6M5bQ+BwHSLUhlvhK8mDjCH?= =?us-ascii?Q?nK2YgD+e7XYo8TE9Up71xkwSLfgx7xMo+yh0wRq0bgWzZTRmkHxa9J7p3CLA?= =?us-ascii?Q?3ySCI2KXFlOUeUr6JDapazIB6pQNN7qlqp5brnDq6dLF1bWzh5pj1WH5QB7l?= =?us-ascii?Q?ymPe0SsXeBX3+JYIGOaed0tW1zAZ4rFxNsmgUJz4FTkKBSOmGLenTaiAQWic?= =?us-ascii?Q?zqRad/nSyfArrKeXARfSKDdcV+j+/yS9pLx1+3vJUI2xfmSuvT6REiR0uCdV?= =?us-ascii?Q?UoMh4pJSokxBC4aIUB1b0Naeh8A/X0SrvnD5Mt7zW4Z+3wgdw/A1XZxPT+Wt?= =?us-ascii?Q?Js8Ddcbcy8QC+2whBQJK+MIWQb1Imj7z1M4Es73UNfAIVBXKUg+yO18B5QLo?= =?us-ascii?Q?A6taFq7f//DLuan9BD40mA0ixb2q8DOBXxjDDl62YLZn43CZEN1JZGvAPr7+?= =?us-ascii?Q?mZJOLRaci5BGlxjo8r/mBWf2LoufZP5hTkYb+5nz95oxCKB6FyyOA+EuCMj8?= =?us-ascii?Q?OO/aGN9QQGG5B1m9MFbRUZctK4f5Vwna6gEgcgxGhyaiAOv7bd3+PcSm1+eK?= =?us-ascii?Q?1E9zJuXUdEgxso3+QULA2ReL1NPzjAc7FPiv6YZsRG3cUMZTHNVh24HZo15j?= =?us-ascii?Q?SxRxSL0Xg3glWxrHnGe8VPWrO92yRTaFsc7fu2XAbPKUY+a7rohmYhYUuLCI?= =?us-ascii?Q?Wx2qq1NYBixItiRuiAC7RthqNPaYyYX3rlox0szoJu4DZjrugAmYky1AXGsc?= =?us-ascii?Q?Ku60cUufQugUNO9x9XbAEIaf81L5fdWmjkPxE32HcUKS+eFVhOLzM37QU8zl?= =?us-ascii?Q?Xf1TEuU+XWOg5gsq7qrDQ+S6Au6IwkS8k+mui7iEnL68RoXV97EjXF3lTi1+?= =?us-ascii?Q?zvQF0xbYV6vX8w5WN08zNg8Jita2tmvNuNuV/zaOVMYr?= Content-Type: multipart/alternative; boundary="_000_LV2PR21MB32536BE011E334AC9AE94041FA289LV2PR21MB3253namp_" MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: LV2PR21MB3253.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: f5b32ea9-3156-42e4-345f-08dab1291108 X-MS-Exchange-CrossTenant-originalarrivaltime: 18 Oct 2022 16:51:56.3716 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: /TKKU1Sb97O51TkAvYybwgAmMTRoPQRBOmnNUYjcG5Wx5wM6SowSksJFogpdSkYKRNebjd4fDC4TC0M4aR81xQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR21MB1999 Subject: [musl] Potential deadlock when fork() is called while __aio_get_queue() is allocating memory --_000_LV2PR21MB32536BE011E334AC9AE94041FA289LV2PR21MB3253namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello, I'm working on the Windows Subsystem Linux, which uses MUSL and I'm current= ly working on solving WSL complete freeze * Issue #8824 * microsoft/WSL (gi= thub.com). I believe that this freeze is caused by a bug causing a deadlock in MUSL. If we look at this sample program: #include #include #include #include #include #include #include void* thread2(void* unused) // Thread 2 { unsigned char buf[256] =3D {0}; struct aiocb aiocb =3D {}; aiocb.aio_fildes =3D open("/proc/self/cmdline", O_RDONLY, S_IRUSR | S_I= WUSR); aiocb.aio_buf =3D buf; aiocb.aio_nbytes =3D sizeof(buf) - 1; aiocb.aio_lio_opcode =3D LIO_READ; if (aio_read(&aiocb) < 0) { printf("aio_read failed, %i\n", errno); return NULL; } int error =3D -1; while(error =3D aio_error(&aiocb) =3D=3D EINPROGRESS) { printf("In progress...\n"); sleep(1); } if (error !=3D 0) { printf("aio_error: %i\n", error); } printf("aio result: %s\n", buf); return NULL; } int main() { pthread_t thread; pthread_create(&thread, NULL, &thread2, NULL); if (fork() > 0) // Thread 1 { pthread_join(thread, NULL); printf("aio complete"); } } Here we have two threads. The main thread is scheduling an asynchronous io = and the other is calling fork(). If we look at what the main thread is doing: * After calling pthread_create, the fork() call goes here, in fork.c. * Before calling _Fork(), this thread will acquire the __malloc_lock (t= hrough __malloc_atfork()), * Then _Fork() is called, which calls __aio_atfork(), which waits for the maploc= k In the meantime, the second thread will: * Call aio_read(), which calls submit(), in aio.c * submit() then calls __aio_get_queue(), which acquires the maplock * At this point the map structure needs to be allocated so submit() cal= ls calloc() * calloc() calls malloc(), in malloc.c * If the allocation overflows, malloc() then calls wrlock(), which wait= s for the __malloc_lock So to summarize: Thread1 holds __malloc_lock and waits for maplock, and thr= ead2 holds maplock and waits for ___malloc_lock. We have a deadlock. This is my understanding of the issue, but I have a (very) limited knowledg= e of MUSL's codebase, so let me know if it's correct. If it is, I think this could be solved by moving __aio_atfork() from _Fork= () to fork(), before __malloc_atfork() so the locks are acquired in the cor= rect order to prevent the deadlock. I do see that _Fork() calls __block_all_sigs before calling __aio_atfork() = so I wonder if the order of these calls is important though (let me know if= that's the case). If this solution makes sense to you, I'm happy to submit a contribution wit= h the fix. Thank you, -- Pierre Boulay --_000_LV2PR21MB32536BE011E334AC9AE94041FA289LV2PR21MB3253namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hello,

 

I’m working on the Windows Subsystem Linux, wh= ich uses MUSL and I’m currently working on solving WSL complete freez= e · Issue #8824 · microsoft/WSL (github.com).<= /p>

 

I believe that this freeze is caused by a bug causin= g a deadlock in MUSL.

 

If we look at this sample program:

 

#include <un= istd.h>

#include <pt= hread.h>

#include <ai= o.h>

#include <fc= ntl.h>

#include <ti= me.h>

#include <st= dio.h>

#include <er= rno.h>

 

void* thread2(v= oid* unused) // Thread 2

{

  &nb= sp; unsigned char buf[256] =3D {0};

  &nb= sp; struct aiocb aiocb =3D {};

  &nb= sp; aiocb.aio_fildes =3D open("/proc/self/cmdline", O_RDONLY, S_I= RUSR | S_IWUSR);

  &nb= sp; aiocb.aio_buf =3D buf;

  &nb= sp; aiocb.aio_nbytes =3D sizeof(buf) - 1;

  &nb= sp; aiocb.aio_lio_opcode =3D LIO_READ;

 

  &nb= sp; if (aio_read(&aiocb) < 0)

  &nb= sp; {

  &nb= sp;     printf("aio_read failed, %i\n", errno= );

  &nb= sp;     return NULL;

  &nb= sp; }

 

  &nb= sp; int error =3D -1;

  &nb= sp; while(error =3D aio_error(&aiocb) =3D=3D EINPROGRESS)

  &nb= sp; {

  &nb= sp;     printf("In progress...\n");

  &nb= sp;     sleep(1);

  &nb= sp; }

 

  &nb= sp; if (error !=3D 0)

  &nb= sp; {

  &nb= sp;     printf("aio_error: %i\n", error);

  &nb= sp; }

 

  &nb= sp; printf("aio result: %s\n", buf);

  &nb= sp; return NULL;

}

 

int main()=

{

  &nb= sp; pthread_t thread;

  &nb= sp; pthread_create(&thread, NULL, &thread2, NULL);

 

  &nb= sp; if (fork() > 0) // Thread 1

  &nb= sp; {

  &nb= sp;     pthread_join(thread, NULL);

  &nb= sp;     printf("aio complete");

  &nb= sp; }

 

}

 

Here we have two threads. The main thread is schedul= ing an asynchronous io and the other is calling fork().

If we look at what the main thread is doing:

 

  • After calling pthread_create, the fork() call goes here, in fork.c.
  • Before calling _Fork(), this thread will acquire the = __malloc_lock (through __malloc_atfork()),
  • Then _Fork() is called, which calls __aio_atfork(), which waits for the ma= plock

 

In the meantime, the second thread will:<= /p>

 

  • Call aio_read(), which calls submit(), in aio.c
  • submit() then calls __aio_get_queue(), which acquires the maplock
  • At this point the map structure needs to be allocated so submi= t() calls calloc()
  • calloc() calls malloc(), in malloc.c
  • If the allocation overflows,= malloc() then calls wrlock(), which waits for the __malloc_lock=

 

So to summarize: Thread1 holds __malloc_lock and wai= ts for maplock, and thread2 holds maplock and waits for ___malloc_lock. We = have a deadlock.

This is my understanding of the issue, but I have a = (very) limited knowledge of MUSL’s codebase, so let me know if itR= 17;s correct.

 

If it is, I think this could be solved by  movi= ng __aio_atfork() from _Fork() to fork(), before __malloc_atfork() so the l= ocks are acquired in the correct order to prevent the deadlock.<= /p>

 

I do see that _Fork() calls __block_all_sigs before = calling __aio_atfork() so I wonder if the order of these calls is important= though (let me know if that’s the case).

If this solution makes sense to you, I’m happy= to submit a contribution with the fix.

 

Thank you,

 

--

Pierre Boulay

 

 

 

 

 

 

 

 

 

 

 

 

 

--_000_LV2PR21MB32536BE011E334AC9AE94041FA289LV2PR21MB3253namp_--