On Sep 26, 2022, at 23:33, Colin Cross <ccross@android.com> wrote:

On Mon, Sep 26, 2022 at 4:12 PM Colin Cross <ccross@android.com> wrote:

On Mon, Sep 26, 2022 at 4:02 PM Rich Felker <dalias@libc.org> wrote:

On Mon, Sep 26, 2022 at 03:42:01PM -0700, Colin Cross wrote:

On Mon, Sep 26, 2022 at 3:38 PM Colin Cross <ccross@android.com> wrote:

On Tue, Aug 23, 2022 at 1:18 AM Szabolcs Nagy <nsz@port70.net> wrote:

* Colin Cross <ccross@android.com> [2022-08-22 17:22:06 -0700]:

On Sat, Aug 20, 2022 at 2:43 AM Szabolcs Nagy <nsz@port70.net> wrote:

i would not use Scrt1.o though, the same toolchain should be
usable for normal linking and relinterp linking, just use a
different name like Xcrt1.o.

Is there some way to get gcc/clang to use Xcrt1.o without using
-nostdlib and passing all the crtbegin/end objects manually?

this requires compiler changes (new cmdline flag) but then i think
the code is upstreamable.

I've used relinterp.o for now, and selected instead of Scrt1.o in
musl-gcc.specs and ld.musl-clang.

i would make Xcrt1.o self-contained and size optimized: it only
runs at start up, this is a different requirement from the -O3
build of normal string functions. and then there is no dependency
on libc internals (which may have various instrumentations that
does not work in Xcrt1.o).

Doesn't this same logic apply to most of the code in dynlink.c?  My
main worry with a self contained implementation is that it requires
reimplementations of various string functions that are easy to get
wrong.  The current prototype reuses the C versions of musl's string
functions, but implements its own syscall wrappers to avoid
interactions with musl internals like errno.

dynlink is in libc.so so it can use code from there.

but moving libc code into the executable has different constraints.
so you will have to make random decisions that string functions are
in but errno is out, wrt which libc internal makes sense in the exe.

i would just keep a separate implementation (or at least compile
the code separately). string functions are easy to implement if
you dont try to optimize them imo. then you have full control over
what is going on in the exe entry code.

I left the reimplementations of string functions and syscalls as
suggested.  Patch attached.

From 0df460188b95f79272003bd0e5c12bceb2a3c25f Mon Sep 17 00:00:00 2001
From: Colin Cross <ccross@android.com>
Date: Thu, 22 Sep 2022 19:14:01 -0700
Subject: [PATCH] Add entry point to find dynamic loader relative to the
executable

Distributing binaries built against musl to systems that don't already
have musl is problematic due to the hardcoded absolute path to the
dynamic loader (e.g. /lib/ld-musl-$ARCH.so.1) in the PT_INTERP header.
This patch adds a feature to avoid the problem by leaving out PT_INTERP
and replacing Scrt1.o with an entry point that can find the dynamic
loader using DT_RUNPATH or LD_LIBRARY_PATH.

The entry point is in crt/relinterp.c.  It uses auxval to get the
program headers and find the load address of the binary, then
searches LD_LIBRARY_PATH or DT_RUNPATH for the dynamic loader.
Once found, it mmaps the loader similar to the way the kernel
does when PT_INTERP is set.  The musl loader uses PT_INTERP to set
the path to the loader in the shared library info exported to the
debugger, so relinterp creates a copy of the program headers
with the PT_INTERP entry added pointing to the found location of
the dynamic loader.  It updates AT_BASE to point to the address
of the dynamic loader, then jumps to the loaders entry point.

The dynamic loader then loads shared libraries and handles
relocations before jumping to the executable's entry point, which is
the entry point in relinterp.c again.  Relinterp detects that
relocations have been performed and calls __libc_start_main, the
same way Scrt1.o would have.

Since relinterp runs before relocations have been performed it has
to avoid referecing any libc functions.  That means reimplementing
the few syscalls and string functions that it uses, and avoiding
implicit calls to memcpy and memset that may  be inserted by the
compiler.

Enabling relinterp is handled in the spec file for gcc and in
the linker script for clang via a -relinterp argument.

Normally gdb and lldb look for a symbol named "_dl_debug_state" in
the interpreter to get notified when the dynamic loader has modified
the list of shared libraries.  When using relinterp the debugger is
not aware of the interpreter (at process launch PT_INTERP is unset
and auxv AT_BASE is 0) so it doesn't know where to look for the symbol.

They fall back to looking in the executable, so we can provide a symbol
in relinterp.c for it to find.  The dynamic loader is then modified
to also find the symbol in the exectuable and to call it from its own
_dl_debug_state function.

The same tests in libc_test pass with or without LDFLAGS += -relinterp
with both musl-gcc and musl-clang.

Ryan Prichard (rprichard@google.com) authored the original prototype
of relinterp.

Have you looked at https://www.openwall.com/lists/musl/2020/03/29/9
where this has already been done? It's not upstream but my
understanding is that the author has been using it successfully for a
long time, and it's been through some review and as I recall was at
least close to acceptable for upstream.

Rich

No, I had not seen that, and it looks to have identical functionality.
I'll experiment with switching to it.

dcrt1 seems to be a perfect drop in replacement for this.  Quick
testing shows it works for my needs on x86_64, x86, aarch64 and arm
architectures.  I've attached a minor patch that fixes some unused
variable and label warnings.
<0001-Fix-unused-variable-and-label-warnings-in-dcrt1.c.patch>