From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-1.2 required=5.0 tests=DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,T_SCC_BODY_TEXT_LINE,URIBL_BLACK autolearn=ham autolearn_force=no version=3.4.4 Received: from second.openwall.net (second.openwall.net [193.110.157.125]) by inbox.vuxu.org (Postfix) with SMTP id A6CA42308D for ; Wed, 7 Feb 2024 19:09:53 +0100 (CET) Received: (qmail 12187 invoked by uid 550); 7 Feb 2024 18:07:07 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 12152 invoked from network); 7 Feb 2024 18:07:06 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707329380; x=1707934180; darn=lists.openwall.com; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=XIOe4jYl5KL19f+sgtV09UBmrIEt1xo6vL8yoluNnt4=; b=AQELr/VTE8e+Ivemp6MMIksNyjwkDeydbgCZNX/DQp28flCYBV8rgFs4/mVEs2YGPx mpRYzwPAbWLu/6IDNYyHYWCQiFq5WGGK8QiqSfoGVLuZbVHmVklfKqsghRtwF/AN5f78 feZg0/lNExIbrg7JC3MWUmxmWtZjKMtWTMIyMFdJ3lChp/Qnk9IlkQwBTX4mJJYqH51h ZkB+o8q6qmkICPEtcMuQf1hIlOk6kmrrUjHAa6ZVp+THCsZm6P9ofin/SaFz39COC0LN HjHOG9Kv7N8MLdD59uhFfJR6ANP33Kejccq2KoWcyVOHHCZ4r0LDRUARxAx+oilA8bhT nR5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707329380; x=1707934180; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XIOe4jYl5KL19f+sgtV09UBmrIEt1xo6vL8yoluNnt4=; b=LD/W4vPNCpzku1cB3leY0uyj9c2iw4waVFetUnog8uC/ql+XTYjgIRXspyQsNdCW60 BjrMkAbyXeAJAyo9rQai+Zmz4sONQDB5RXk1VmttRICtTTFPswABHeidPBrl5fD/Tf44 K0Zv4jvJErHyBKzgDMlxFZOlHQKFJTSiQTWLVKoefMjulKjpN6SgUVYo+YS25V/4zHLf SijQe+PfWDCzPxbziAUyjZK5WMnqKq/UwO5enXy+BCsCNQYl+Nuku6a7jO1YOK7cmTAO mDex7L/qW4N1RKENu1FczvmZUkPVsmU81IaOIEqnqeysvRkN8twnfjGYUFfjlPS2awRn pqQQ== X-Gm-Message-State: AOJu0YyNWAIt4UNpDXZwZ5pLvXWGnu9YMtQQDg77xnuGjcsl+Pp26dJD RoXD5ePF/5/wIM+qb9M2Myuh38xV+CmXLkN2zooGzvzeuaQGyITjXCIKmPw0MKqTlZLU3+9nZFE baIMV1X8V0bOO0e5pFdvTqhUmFuWwBosMUT6lVtxPTOUgJbXwB2lV X-Google-Smtp-Source: AGHT+IHXBTWgn2FEtj9yOBlyn33wyMmPlfb/A34EUTT6jvkp/pwYFyzR/gWa52dN1Y2OJYFeedMuEcNvHSqhd20YiBI= X-Received: by 2002:a17:906:e284:b0:a38:4c32:9a3a with SMTP id gg4-20020a170906e28400b00a384c329a3amr3587175ejb.2.1707329379950; Wed, 07 Feb 2024 10:09:39 -0800 (PST) MIME-Version: 1.0 References: <20240207012247.1121273-1-mmayer@broadcom.com> In-Reply-To: <20240207012247.1121273-1-mmayer@broadcom.com> From: Colin Cross Date: Wed, 7 Feb 2024 10:09:27 -0800 Message-ID: To: musl@lists.openwall.com Cc: Markus Mayer Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [musl] [PATCH 0/1] ldso: continue searching if wrong architecture is found On Tue, Feb 6, 2024 at 5:23=E2=80=AFPM Markus Mayer w= rote: > > Hi all, > > We have just discovered an interesting issue after recently > transitioning from glibc to musl-libc. > > This is an Aarch64 platform with 64-bit userland that also supports > running 32-bit applications. We are using GCC 12.3 and musl 1.2.3. > > Some application teams are still developing their apps as 32-bit > applications. As mentioned, the base system is 64-bit. To ensure all app > dependencies are met, all shared libraries the app needs are bundled > with the app in a tar-ball by the application developers. The launch > script of the app will set LD_LIBRARY_PATH to the directory containing > these 32-bit libraries before calling the actual application. This way, > the app can find all its shared libraries. Some of these 32-bit > libraries are "standard" libraries (libz, libcrypto) that also exist as > 64-bit version in /usr/lib. > > What we have discovered is that 64-bit applications launched by the > 32-bit app will fail due to a shared library mismatch. The 32-bit app > needs to call system utilities, which are 64-bit. So, this needs to > work. > > It took some digging, but I think I know what is going on. Let me > summarize how to reproduce the problem, why it occurs and, hopefully, > what can be done about it. > > The ingredients: > > * A 64-bit system > * A directory containing 32-bit versions of "standard" libraries also > present on the system as 64-bit versions (libz, libcrypto, etc.) > * LD_LIBRARY_PATH pointing to the custom 32-bit library directory > * Attempting to launch a 64-bit application that uses one of the > libraries that exist as 32-bit and 64-bit version > > The problem: > > If LD_LIBRARY_PATH is set to a directory containing 32-bit libraries and > then a 64-bit binary is invoked, the shared library loader will pick up > the 32-bit version of a library first, because it'll look at > LD_LIBRARY_PATH before anything else. Mapping the 32-bit library into > the 64-bit process will fail. This much is expected. > > However, even though the correct library resides on the system, the > shared library loader never attempts to look for it. The 64-bit process > will fail to launch, even though there is no reason for the failure. The > problem only exists, because the shared library launcher doesn't look in > the remaining shared library directories. > > The solution: > > The shared library loader needs to keep searching the rest of the > library search path if the library it found in LD_LIBRARY_PATH could not > be mapped. If the library loader does this, everything will work fine as > long as the library resides on the system in a well known path. > > How to reproduce: > > The problem can be simulated easily in the shell as follows. > > 1) Baseline: call /sbin/lsmod normally. Everything is working. > > $ /sbin/lsmod > Module Size Used by > bdc 53248 0 > udc_core 49152 1 bdc > > 2) Set LD_LIBRARY_PATH to the 32-bit directory and try again. > > $ LD_LIBRARY_PATH=3D/path/to/app /sbin/lsmod > Error loading shared library libz.so.1: Exec format error (needed by /sbi= n/lsmod) > Error loading shared library libcrypto.so.3: Exec format error (needed by= /sbin/lsmod) > > Suddenly the 64-bit binary fails to run, because the copies of libz and > libcrypto the shared library loader finds are the 32-bit versions > residing in the app directory. It never tries looking in /usr/lib. > > Potential solution (the proposed patch): > > # LD_LIBRARY_PATH=3D/path/to/app /path/to/custom/libc.so /sbin/lsmod > Module Size Used by > bdc 53248 0 > udc_core 49152 1 bdc > > With the attached patch applied, everything is working again. Here, I am > still setting LD_LIBRARY_PATH to the offending directory, but then I am > using a patched version of musl-libc to launch /sbin/lsmod. This version > of libc.so contains the patch, so it *will* search the rest of the > system directories after discovering that the 32-bit versions of libz > and libcrypto didn't work out. So, /sbin/lsmod is able to run fine. > > We can confirm this using strace: > > # LD_LIBRARY_PATH=3D. /path/to/custom/libc.so \ > /usr/bin/strace -e openat /path/to/custom/libc.so --list /sbin/lsmod > openat(AT_FDCWD, "/sbin/lsmod", O_RDONLY|O_LARGEFILE) =3D 3 > /lib/ld-musl-aarch64.so.1 (0x7fb72b1000) > openat(AT_FDCWD, "./liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D -1= ENOENT (No such file or directory) > openat(AT_FDCWD, "/etc/ld-musl-aarch64.path", O_RDONLY|O_LARGEFILE|O_CLOE= XEC) =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D= -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/local/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CL= OEXEC) =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC)= =3D 3 > liblzma.so.5 =3D> /usr/lib/liblzma.so.5 (0x7fb7265000) > openat(AT_FDCWD, "./libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D 3 > openat(AT_FDCWD, "/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D -1= ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/local/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEX= EC) =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/lib/libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = =3D 3 > libz.so.1 =3D> /usr/lib/libz.so.1 (0x7fb7251000) > openat(AT_FDCWD, "./libcrypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D = 3 > openat(AT_FDCWD, "/lib/libcrypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/local/lib/libcrypto.so.3", O_RDONLY|O_LARGEFILE|O_= CLOEXEC) =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/lib/libcrypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXE= C) =3D 3 > libcrypto.so.3 =3D> /usr/lib/libcrypto.so.3 (0x7fb6e92000) > libc.so =3D> /lib/ld-musl-aarch64.so.1 (0x7fb72b1000) > > This call to strace is invoking the patched libc.so twice. Once, so > strace itself won't fail, and once to launch /sbin/lsmod. We can see it > finds the 32-bit versions of a number of libraries, but then keeps > searching, once it finds it is unable to map them. In the end, it finds > the proper libraries, and everything is working. > > Conversely, the unpatched libc.so will not try any other location > besides the one pointed to by LD_LIBRARY_PATH, if library it is looking > for exists in LD_LIBRARY_PATH. (It'll find the proper liblzma, because > that does NOT exist as 32-bit version in LD_LIBRARY_PATH. But libz and > libcrypto do, and that's where it fails.) > > openat(AT_FDCWD, "/sbin/lsmod", O_RDONLY|O_LARGEFILE) =3D 3 > openat(AT_FDCWD, "./liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D -1= ENOENT (No such file or directory) > openat(AT_FDCWD, "/etc/ld-musl-aarch64.path", O_RDONLY|O_LARGEFILE|O_CLOE= XEC) =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D= -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/local/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CL= OEXEC) =3D -1 ENOENT (No such file or directory) > openat(AT_FDCWD, "/usr/lib/liblzma.so.5", O_RDONLY|O_LARGEFILE|O_CLOEXEC)= =3D 3 > openat(AT_FDCWD, "./libz.so.1", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D 3 > openat(AT_FDCWD, "./libcrypto.so.3", O_RDONLY|O_LARGEFILE|O_CLOEXEC) =3D = 3 > +++ exited with 127 +++ > > My proposal is, of course, only one possibility of many to achieve the > goal of searching the system library locations after mapping a library > from LD_LIBRARY_PATH fails. > > What is your take? Does the idea make sense in principal? Does my patch > make sense? > > Continuing to search the system directories does seem to be the right > thing to do under the circumstances described here. Also, it is what > glibc does. > > Regards, > -Markus > > Markus Mayer (1): > ldso: continue searching if wrong architecture is found first > > ldso/dynlink.c | 18 ++++++++++++++++-- > 1 file changed, 16 insertions(+), 2 deletions(-) > > -- > 2.43.0 > There is a previous discussion of the same issue at https://www.openwall.com/lists/musl/2023/02/07/3.