From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=0.6 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, T_SCC_BODY_TEXT_LINE,URIBL_BLACK autolearn=no autolearn_force=no version=3.4.4 Received: (qmail 6304 invoked from network); 23 Aug 2022 00:22:36 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 23 Aug 2022 00:22:36 -0000 Received: (qmail 4084 invoked by uid 550); 23 Aug 2022 00:22:32 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 4046 invoked from network); 23 Aug 2022 00:22:31 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20210112; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc; bh=r9+pa4upJrL3PhSH6ZqgIujTW7JMnGEwwzRVH+6plkU=; b=WJcFpAfKgc6wHccWb34ig3IiA806Bx3W08XFHk1NhDKVvbwgenTJj/DkCN/nkgotWy UCtHAJ8P59mqLmYgpp+oH314zUnc5y49cHJWI3Bjr+ckfYfOJHM0i69OqoEc1s4MYC8U qM/P5cmHujPjqzdscSKrS5pLuUZRBNRKbdXMlLI4hgET9B1CpUoAqtAH78oSub2HV4U9 5kOGhOg/0DaqdAAIyVWSu1uNJlAOG5KQlnX+d0pnfUOV+M3mAZjaOf5tU4ru/9Gm08gK P0vw/KSBv7rNQOe4dAN0BHFmptml2SrZMpLp2/RX2b7n9efLNjjDYh/y/ri99MZC8UXn t0pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc; bh=r9+pa4upJrL3PhSH6ZqgIujTW7JMnGEwwzRVH+6plkU=; b=bEEv12HiUH9FuFKYFYK9fsQK91eaaI7wxOIi8XSXV3Xuv3LH1pSRQHEXfq9rY9V1QA cCb+bUi+MH9N4ZVDjj5TErFjZ0Y/wGL8w+oYspiiNrsLjCbA2nffcqjEhpO6jfMbqGof u0ruXD9fH0J5devixfJgjm6KL7Tf8UE/i5ON/Uamn5UPvYQPl9SajICRqfGyM84+3TB1 afuM5SbvPvqoDczYIuE1lB+y2h+KavbNX5F8NB5Pd488tiz9HWiUtfd1qeKfH+C4Mf5r tT8egWxUsa5LHXF3DUmvcrXRxFv6MVgfuH7NAPrM79mkFh0j2NbvbV4+CLJMlM7yDFz4 O42Q== X-Gm-Message-State: ACgBeo0eaIUfwRdUKKLiB+hhXWabmn77XijkTf19q+PpmvHzq+3DkF0P 5MbFKuZsw+W/XdP9Hy5AxqffYE+UoeR+PHw6qPIsZA== X-Google-Smtp-Source: AA6agR4IG2zwWD2bEaAzoa0HrFytpgzKYc8QtK6bLRECBEyeF2JLHz0EkYRfxGcvJ9iFkkCM+kxwp90Rq+AEwrRKaSU= X-Received: by 2002:a05:6602:2be6:b0:688:c977:92d2 with SMTP id d6-20020a0566022be600b00688c97792d2mr9732290ioy.179.1661214138933; Mon, 22 Aug 2022 17:22:18 -0700 (PDT) MIME-Version: 1.0 References: <20220820094308.GK1320090@port70.net> In-Reply-To: <20220820094308.GK1320090@port70.net> From: Colin Cross Date: Mon, 22 Aug 2022 17:22:06 -0700 Message-ID: To: Colin Cross , musl@lists.openwall.com, Ryan Prichard Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [musl] Running musl executables without a preinstalled dynamic linker On Sat, Aug 20, 2022 at 2:43 AM Szabolcs Nagy wrote: > > * Colin Cross [2022-08-15 14:35:33 -0700]: > > I would like to distribute dynamic binaries built against musl to > > systems that do not have the musl dynamic linker installed in any > > known location (e.g. /lib/ld-musl-$ARCH.so.1). I have two prototypes > > that enable this, and I=E2=80=99d like to gauge whether either is somet= hing > > that would be of interest to check in to musl, or whether it would be > > something we should keep in our project. > > > > The first solution is based on the embedded linker we use to test > > bionic libc on non-Android systems. The dynamic linker is compiled as > > usual, then the resulting elf file is embedded as raw data into > > Scrt1.o and the PT_INTERP section removed. The entry point is changed > > to point to a trampoline that modifies AT_BASE, AT_ENTRY and AT_PHDR > > to simulate how the kernel would initialize them if the dynamic linker > > was mapped separately by the kernel instead of as part of the main > > executable, and then jumps to the dynamic linker. > > > > This embedded linker solution works relatively well, except that the > > dynamic linker=E2=80=99s elf sections are inside the main executable=E2= =80=99s elf > > sections, which can break reasonable assumptions. For example, musl=E2= =80=99s > > dladdr fails to find symbols in the embedded linker, and gdb has > > trouble finding debug information from the linker. Musl=E2=80=99s reus= e of > > libc.so as the linker means that these problems apply to everything in > > libc.so, and also increases the size of every binary by including all > > of libc.so. > > > > These problems with the embedded linker could be somewhat mitigated by > > splitting the dynamic linker out of libc.so when using the embedded > > linker. That requires compiling the ldso sources against a statically > > linked libc.a, tweaking some of the initialization, and forwarding the > > dl* calls from libc.so to the separate linker. The changes are > > relatively small, but result in a pretty big difference in musl=E2=80= =99s > > internals with and without the embedded linker that may be hard to > > maintain. > > > > that breaks atomic update of the libc and introduces libc internal abi. > (i.e. bad for long term security and maintainability) The intent was to distribute the binary with the embedded linker alongside a matching copy of libc, but yes, this would increase the chances of an unintended version skew. > > The second solution we call =E2=80=9Crelinterp=E2=80=9D. It was origin= ally designed > > by Ryan Prichard as a standalone trampoline that could be used with > > musl, glibc or bionic, but I=E2=80=99ve more tightly integrated it with= musl > > in order to reuse CRTJMP for architecture portability and some of > > musl=E2=80=99s string functions to reduce the size of the code. It use= s a > > similar trampoline in Scrt1.o, but with a much larger implementation > > that reads DT_RUNPATH to construct a path to the dynamic linker that > > is relative to the executable. It then maps the dynamic linker as the > > kernel would, modifies AT_BASE, AT_ENTRY and AT_PHDR, and jumps to the > > dynamic linker. > > > > i think this is a better approach. Agreed, I generally prefer this approach. > i would not use Scrt1.o though, the same toolchain should be > usable for normal linking and relinterp linking, just use a > different name like Xcrt1.o. Is there some way to get gcc/clang to use Xcrt1.o without using -nostdlib and passing all the crtbegin/end objects manually? > > The current prototype of relinterp is tricky to compile, as it > > requires using -fvisibility=3Dhidden and ld -r partial linking to build > > a Scrt1.o file that uses some of the src/string/*.c sources without > > any relocations, and then objcopy =E2=80=93keep-global-symbol to hide t= he > > string symbols. It=E2=80=99s only useful if DT_RUNPATH contains $ORIGI= N so > > that the dynamic linker can be distributed alongside the executable, > > so it is probably never going to be suitable for setuid binaries. > > > > If relinterp were going to be included with musl I=E2=80=99d refactor i= t to > > reuse the __dls* bootstrapping from dynlink.c so that it can link > > against libc.a and not worry about avoiding any relocations. > > > > i would make Xcrt1.o self-contained and size optimized: it only > runs at start up, this is a different requirement from the -O3 > build of normal string functions. and then there is no dependency > on libc internals (which may have various instrumentations that > does not work in Xcrt1.o). Doesn't this same logic apply to most of the code in dynlink.c? My main worry with a self contained implementation is that it requires reimplementations of various string functions that are easy to get wrong. The current prototype reuses the C versions of musl's string functions, but implements its own syscall wrappers to avoid interactions with musl internals like errno. > > An alternative solution to these two would be to distribute statically > > linked binaries, which precludes the use of dlopen, or to wrap every > > executable in a shell script that runs the dynamic linker directly. > > > > i think it is possible to support static linking such that if dlopen > is linked then the entire libc gets linked into the main exe with > libc apis exported. then dlopen can work from an otherwise static exe. > (may not be easy to implement in practice though) > > > Do either of these prototypes seem interesting enough to clean up and > > post as upstream patches, or should I keep them as a side project that > > I can bolt on to musl with minimal invasive changes? > > > > Colin