From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=0.6 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HTML_MESSAGE,MAILING_LIST_MULTI,RCVD_IN_MSPIKE_H2, URIBL_BLACK autolearn=no autolearn_force=no version=3.4.4 Received: (qmail 22975 invoked from network); 27 Sep 2022 05:09:32 -0000 Received: from second.openwall.net (193.110.157.125) by inbox.vuxu.org with ESMTPUTF8; 27 Sep 2022 05:09:32 -0000 Received: (qmail 17735 invoked by uid 550); 27 Sep 2022 05:09:29 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 17678 invoked from network); 27 Sep 2022 05:09:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rcombs.me; s=google; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date; bh=hOZR4wEjS9sO4FR7/kWDmtthPMo/Twn62YI596KLDww=; b=Wf2YuN4JHSFigUFEBNApB3kB3B4w6oh4QrCpe4OnSIb94WtXbd7LmLrMy68sY1gNJF lh+U5lbxFNtMFWrHVU4i5ERGvxAYJX4v8dbQ9BgdGsHttfl4zs+eCG4YKGWHbwSqG+EU zjNSYDa7WDMcdqqk+h3/pTe8Omid9H0WTnWgQHlfsYQfMuPZXnzeDBVPnPVez0sS3l65 2iTnsUTfSVa62umGi+YdErZ1/evSixW0pF1JxGzRGpq6pwzhYXBamSZT202mcuFWrZoR OX8pgfreA7MdfD3m1XdQehaHBW4bSmIiYJm7+gpfSXmJ3Psp3/UKuxD7to1++ds5m0SU rBeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date; bh=hOZR4wEjS9sO4FR7/kWDmtthPMo/Twn62YI596KLDww=; b=653FVeO5RA5ADKFdF/o9LqyjB2CivK34ncmKh7vd+24qULadqHxkU/bQdmBY42W8Vb WW0uh9G0YeeyOX0jAjhZnUuxbW3V0tMKGeQ3MSs0JFMkp6qTd99BDJbJvQhqfae8Nrvh b5+XeEkt7GAzzigebIL/WxZjAdtXkXinvSgw17W6RPC6DgXHQa9z2qhL0iG63fchsYjn E4pyWUJ1A+S0k42T9UsvnKvmZ65oaCqBZAPYmvFi6F2+d5ldRINdQZjAAOTirY24g0bU +GEsx8Q5CDRj8W1/ha5c2mjj8Hea9a5dtPKPxyLibHYvSzWzKlyKWga8h7wOAGlU7Y1P 4QNg== X-Gm-Message-State: ACrzQf0+ZJoFAuCrycmPxSVeXKDDWkGi1vA0QyX+5VxGqcegN1zteyXF x9CSPRpfuJSTB5hQrcVLvJUm1Z6UfjVK X-Google-Smtp-Source: AMsMyM7e6pZ9toc+TlKS/fXITz+vA53q7Gl92f189Y/YLj0212SPUGHT0SrqiebGf6DOe6eO11k8Yg== X-Received: by 2002:a05:6602:2dcb:b0:68b:7804:e924 with SMTP id l11-20020a0566022dcb00b0068b7804e924mr10958598iow.54.1664255355944; Mon, 26 Sep 2022 22:09:15 -0700 (PDT) From: Ridley Combs Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_A2890DA7-D664-47D2-A55B-710850737144" Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Date: Tue, 27 Sep 2022 00:09:11 -0500 In-Reply-To: Cc: Rich Felker , Ryan Prichard , Colin Cross To: musl@lists.openwall.com References: <20220820094308.GK1320090@port70.net> <20220823081802.GM1320090@port70.net> <20220926230241.GF9709@brightrain.aerifal.cx> X-Mailer: Apple Mail (2.3696.120.41.1.1) Subject: Re: [musl] Running musl executables without a preinstalled dynamic linker --Apple-Mail=_A2890DA7-D664-47D2-A55B-710850737144 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > On Sep 26, 2022, at 23:33, Colin Cross wrote: >=20 > On Mon, Sep 26, 2022 at 4:12 PM Colin Cross = wrote: >>=20 >> On Mon, Sep 26, 2022 at 4:02 PM Rich Felker wrote: >>>=20 >>> On Mon, Sep 26, 2022 at 03:42:01PM -0700, Colin Cross wrote: >>>> On Mon, Sep 26, 2022 at 3:38 PM Colin Cross = wrote: >>>>>=20 >>>>> On Tue, Aug 23, 2022 at 1:18 AM Szabolcs Nagy = wrote: >>>>>>=20 >>>>>> * Colin Cross [2022-08-22 17:22:06 -0700]: >>>>>>> On Sat, Aug 20, 2022 at 2:43 AM Szabolcs Nagy = wrote: >>>>>>>> i would not use Scrt1.o though, the same toolchain should be >>>>>>>> usable for normal linking and relinterp linking, just use a >>>>>>>> different name like Xcrt1.o. >>>>>>>=20 >>>>>>> Is there some way to get gcc/clang to use Xcrt1.o without using >>>>>>> -nostdlib and passing all the crtbegin/end objects manually? >>>>>>=20 >>>>>> this requires compiler changes (new cmdline flag) but then i = think >>>>>> the code is upstreamable. >>>>>=20 >>>>> I've used relinterp.o for now, and selected instead of Scrt1.o in >>>>> musl-gcc.specs and ld.musl-clang. >>>>>=20 >>>>>>=20 >>>>>>>> i would make Xcrt1.o self-contained and size optimized: it only >>>>>>>> runs at start up, this is a different requirement from the -O3 >>>>>>>> build of normal string functions. and then there is no = dependency >>>>>>>> on libc internals (which may have various instrumentations that >>>>>>>> does not work in Xcrt1.o). >>>>>>>=20 >>>>>>> Doesn't this same logic apply to most of the code in dynlink.c? = My >>>>>>> main worry with a self contained implementation is that it = requires >>>>>>> reimplementations of various string functions that are easy to = get >>>>>>> wrong. The current prototype reuses the C versions of musl's = string >>>>>>> functions, but implements its own syscall wrappers to avoid >>>>>>> interactions with musl internals like errno. >>>>>>=20 >>>>>> dynlink is in libc.so so it can use code from there. >>>>>>=20 >>>>>> but moving libc code into the executable has different = constraints. >>>>>> so you will have to make random decisions that string functions = are >>>>>> in but errno is out, wrt which libc internal makes sense in the = exe. >>>>>>=20 >>>>>> i would just keep a separate implementation (or at least compile >>>>>> the code separately). string functions are easy to implement if >>>>>> you dont try to optimize them imo. then you have full control = over >>>>>> what is going on in the exe entry code. >>>>>=20 >>>>> I left the reimplementations of string functions and syscalls as >>>>> suggested. Patch attached. >>>=20 >>>> =46rom 0df460188b95f79272003bd0e5c12bceb2a3c25f Mon Sep 17 00:00:00 = 2001 >>>> From: Colin Cross >>>> Date: Thu, 22 Sep 2022 19:14:01 -0700 >>>> Subject: [PATCH] Add entry point to find dynamic loader relative to = the >>>> executable >>>>=20 >>>> Distributing binaries built against musl to systems that don't = already >>>> have musl is problematic due to the hardcoded absolute path to the >>>> dynamic loader (e.g. /lib/ld-musl-$ARCH.so.1) in the PT_INTERP = header. >>>> This patch adds a feature to avoid the problem by leaving out = PT_INTERP >>>> and replacing Scrt1.o with an entry point that can find the dynamic >>>> loader using DT_RUNPATH or LD_LIBRARY_PATH. >>>>=20 >>>> The entry point is in crt/relinterp.c. It uses auxval to get the >>>> program headers and find the load address of the binary, then >>>> searches LD_LIBRARY_PATH or DT_RUNPATH for the dynamic loader. >>>> Once found, it mmaps the loader similar to the way the kernel >>>> does when PT_INTERP is set. The musl loader uses PT_INTERP to set >>>> the path to the loader in the shared library info exported to the >>>> debugger, so relinterp creates a copy of the program headers >>>> with the PT_INTERP entry added pointing to the found location of >>>> the dynamic loader. It updates AT_BASE to point to the address >>>> of the dynamic loader, then jumps to the loaders entry point. >>>>=20 >>>> The dynamic loader then loads shared libraries and handles >>>> relocations before jumping to the executable's entry point, which = is >>>> the entry point in relinterp.c again. Relinterp detects that >>>> relocations have been performed and calls __libc_start_main, the >>>> same way Scrt1.o would have. >>>>=20 >>>> Since relinterp runs before relocations have been performed it has >>>> to avoid referecing any libc functions. That means reimplementing >>>> the few syscalls and string functions that it uses, and avoiding >>>> implicit calls to memcpy and memset that may be inserted by the >>>> compiler. >>>>=20 >>>> Enabling relinterp is handled in the spec file for gcc and in >>>> the linker script for clang via a -relinterp argument. >>>>=20 >>>> Normally gdb and lldb look for a symbol named "_dl_debug_state" in >>>> the interpreter to get notified when the dynamic loader has = modified >>>> the list of shared libraries. When using relinterp the debugger is >>>> not aware of the interpreter (at process launch PT_INTERP is unset >>>> and auxv AT_BASE is 0) so it doesn't know where to look for the = symbol. >>>>=20 >>>> They fall back to looking in the executable, so we can provide a = symbol >>>> in relinterp.c for it to find. The dynamic loader is then modified >>>> to also find the symbol in the exectuable and to call it from its = own >>>> _dl_debug_state function. >>>>=20 >>>> The same tests in libc_test pass with or without LDFLAGS +=3D = -relinterp >>>> with both musl-gcc and musl-clang. >>>>=20 >>>> Ryan Prichard (rprichard@google.com) authored the original = prototype >>>> of relinterp. >>>=20 >>> Have you looked at https://www.openwall.com/lists/musl/2020/03/29/9 >>> where this has already been done? It's not upstream but my >>> understanding is that the author has been using it successfully for = a >>> long time, and it's been through some review and as I recall was at >>> least close to acceptable for upstream. >>>=20 >>> Rich >>=20 >> No, I had not seen that, and it looks to have identical = functionality. >> I'll experiment with switching to it. >=20 > dcrt1 seems to be a perfect drop in replacement for this. Quick > testing shows it works for my needs on x86_64, x86, aarch64 and arm > architectures. I've attached a minor patch that fixes some unused > variable and label warnings. > <0001-Fix-unused-variable-and-label-warnings-in-dcrt1.c.patch> Here's the current version of the patch from my fork; it already has the = changes you mentioned: = https://github.com/rcombs/musl/commit/740155e21f7057a33e75a9ed4cb6fbf07f75= d2a7 = Also notable, I have in fact tested this against glibc and it works fine = with that as well; I'd imagine there's a decent chance it'd also work = with bionic. I haven't kept up with upstreaming it since it's fairly self-contained = and unlikely to regularly have substantial conflicts with upstream, so = it's far easier to just rebase a fork now and then than to deal with = mailing-list crap, but if anybody wants to merge this, go right on = ahead; I'll be happy to answer any questions or discuss any improvements = on IRC (rcombs on the relevant networks), Discord (rcombs#1111), Twitter = (@11rcombs), GitHub (rcombs), or wherever else is convenient. If none of = those work, feel free to reach out to me directly at this address and we = can figure something out. I won't be sending any additional emails on this thread (the = sender-forging the listserv will do after I send this one alone is = already going to subject me to plenty of DMARC rejection notices, thank = you very much). Again, happy to help in any way I can, just not via = mailing list. I'm making an exception this once since you reached out = directly about something I'd worked on, but in general I do my best not = to interact with these things unless somebody's paying me to. --Ridley --Apple-Mail=_A2890DA7-D664-47D2-A55B-710850737144 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Sep 26, 2022, at 23:33, Colin Cross <ccross@android.com> = wrote:

On Mon, Sep 26, 2022 at 4:12 PM Colin Cross <ccross@android.com> = wrote:

On Mon, Sep 26, 2022 at 4:02 PM Rich Felker <dalias@libc.org> = wrote:

On Mon, Sep 26, 2022 at 03:42:01PM -0700, Colin Cross = wrote:
On Mon, Sep = 26, 2022 at 3:38 PM Colin Cross <ccross@android.com> wrote:

On Tue, Aug 23, 2022 at 1:18 AM = Szabolcs Nagy <nsz@port70.net> wrote:

* Colin Cross <ccross@android.com> = [2022-08-22 17:22:06 -0700]:
On Sat, Aug 20, 2022 at 2:43 AM Szabolcs Nagy <nsz@port70.net> = wrote:
i would not = use Scrt1.o though, the same toolchain should be
usable = for normal linking and relinterp linking, just use a
different name like Xcrt1.o.

Is there some way to get gcc/clang to use Xcrt1.o without = using
-nostdlib and passing all the crtbegin/end objects = manually?

this requires = compiler changes (new cmdline flag) but then i think
the = code is upstreamable.

I've = used relinterp.o for now, and selected instead of Scrt1.o in
musl-gcc.specs and ld.musl-clang.


i would = make Xcrt1.o self-contained and size optimized: it only
runs= at start up, this is a different requirement from the -O3
build of normal string functions. and then there is no = dependency
on libc internals (which may have various = instrumentations that
does not work in Xcrt1.o).

Doesn't this same logic apply to = most of the code in dynlink.c?  My
main worry with a = self contained implementation is that it requires
reimplementations of various string functions that are easy = to get
wrong.  The current prototype reuses the C = versions of musl's string
functions, but implements its = own syscall wrappers to avoid
interactions with musl = internals like errno.

dynlink = is in libc.so so it can use code from there.

but moving libc code into the executable has different = constraints.
so you will have to make random decisions = that string functions are
in but errno is out, wrt which = libc internal makes sense in the exe.

i = would just keep a separate implementation (or at least compile
the code separately). string functions are easy to implement = if
you dont try to optimize them imo. then you have full = control over
what is going on in the exe entry code.

I left the reimplementations of = string functions and syscalls as
suggested.  Patch = attached.

=46rom = 0df460188b95f79272003bd0e5c12bceb2a3c25f Mon Sep 17 00:00:00 2001
From: Colin Cross <ccross@android.com>
Date: Thu, 22 Sep = 2022 19:14:01 -0700
Subject: [PATCH] Add entry point to = find dynamic loader relative to the
executable

Distributing binaries built against musl to = systems that don't already
have musl is problematic due to = the hardcoded absolute path to the
dynamic loader (e.g. = /lib/ld-musl-$ARCH.so.1) in the PT_INTERP header.
This = patch adds a feature to avoid the problem by leaving out PT_INTERP
and replacing Scrt1.o with an entry point that can find the = dynamic
loader using DT_RUNPATH or LD_LIBRARY_PATH.

The entry point is in crt/relinterp.c. =  It uses auxval to get the
program headers and find = the load address of the binary, then
searches = LD_LIBRARY_PATH or DT_RUNPATH for the dynamic loader.
Once = found, it mmaps the loader similar to the way the kernel
does when PT_INTERP is set.  The musl loader uses = PT_INTERP to set
the path to the loader in the shared = library info exported to the
debugger, so relinterp = creates a copy of the program headers
with the PT_INTERP = entry added pointing to the found location of
the dynamic = loader.  It updates AT_BASE to point to the address
of = the dynamic loader, then jumps to the loaders entry point.

The dynamic loader then loads shared libraries = and handles
relocations before jumping to the executable's = entry point, which is
the entry point in relinterp.c = again.  Relinterp detects that
relocations have been = performed and calls __libc_start_main, the
same way = Scrt1.o would have.

Since relinterp runs = before relocations have been performed it has
to avoid = referecing any libc functions.  That means reimplementing
the few syscalls and string functions that it uses, and = avoiding
implicit calls to memcpy and memset that may =  be inserted by the
compiler.

Enabling relinterp is handled in the spec file for gcc and = in
the linker script for clang via a -relinterp = argument.

Normally gdb and lldb look for a = symbol named "_dl_debug_state" in
the interpreter to get = notified when the dynamic loader has modified
the list of = shared libraries.  When using relinterp the debugger is
not aware of the interpreter (at process launch PT_INTERP is = unset
and auxv AT_BASE is 0) so it doesn't know where to = look for the symbol.

They fall back to = looking in the executable, so we can provide a symbol
in = relinterp.c for it to find.  The dynamic loader is then modified
to also find the symbol in the exectuable and to call it from = its own
_dl_debug_state function.

The same tests in libc_test pass with or without LDFLAGS +=3D = -relinterp
with both musl-gcc and musl-clang.

Ryan Prichard (rprichard@google.com) = authored the original prototype
of relinterp.

Have you looked at https://www.openwall.com/lists/musl/2020/03/29/9
where this has already been done? It's not upstream but my
understanding is that the author has been using it = successfully for a
long time, and it's been through some = review and as I recall was at
least close to acceptable = for upstream.

Rich

No, I had not seen that, and it = looks to have identical functionality.
I'll experiment = with switching to it.

dcrt1 = seems to be a perfect drop in replacement for this.  Quick
testing shows it works for my needs on x86_64, x86, aarch64 = and arm
architectures.  I've attached a minor patch = that fixes some unused
variable and label warnings.
<0001-Fix-unused-variable-and-label-warnings-in-= dcrt1.c.patch>

Here's the current version of the patch from my = fork; it already has the changes you mentioned: https://github.com/rcombs/musl/commit/740155e21f7057a33e75a9ed4= cb6fbf07f75d2a7

Also notable, I = have in fact tested this against glibc and it works fine with that as = well; I'd imagine there's a decent chance it'd also work with = bionic.

I haven't kept up with = upstreaming it since it's fairly self-contained and unlikely to = regularly have substantial conflicts with upstream, so it's far easier = to just rebase a fork now and then than to deal with mailing-list crap, = but if anybody wants to merge this, go right on ahead; I'll be happy to = answer any questions or discuss any improvements on IRC (rcombs on the = relevant networks), Discord (rcombs#1111), Twitter (@11rcombs), GitHub = (rcombs), or wherever else is convenient. If none of those work, feel = free to reach out to me directly at this address and we can figure = something out.

I won't be sending = any additional emails on this thread (the sender-forging the listserv = will do after I send this one alone is already going to subject me to plenty of DMARC rejection notices, thank you very much). = Again, happy to help in any way I can, just not via mailing list. I'm = making an exception this once since you reached out directly about = something I'd worked on, but in general I do my best not to interact = with these things unless somebody's paying me to.

--Ridley

= --Apple-Mail=_A2890DA7-D664-47D2-A55B-710850737144--