Re: [musl] Running musl executables without a preinstalled dynamic linker

[Szabolcs Nagy <nsz@port70.net>]

* Colin Cross <ccross@android.com> [2022-08-15 14:35:33 -0700]:
> I would like to distribute dynamic binaries built against musl to
> systems that do not have the musl dynamic linker installed in any
> known location (e.g. /lib/ld-musl-$ARCH.so.1).  I have two prototypes
> that enable this, and I’d like to gauge whether either is something
> that would be of interest to check in to musl, or whether it would be
> something we should keep in our project.
> 
> The first solution is based on the embedded linker we use to test
> bionic libc on non-Android systems.  The dynamic linker is compiled as
> usual, then the resulting elf file is embedded as raw data into
> Scrt1.o and the PT_INTERP section removed.  The entry point is changed
> to point to a trampoline that modifies AT_BASE, AT_ENTRY and AT_PHDR
> to simulate how the kernel would initialize them if the dynamic linker
> was mapped separately by the kernel instead of as part of the main
> executable, and then jumps to the dynamic linker.
> 
> This embedded linker solution works relatively well, except that the
> dynamic linker’s elf sections are inside the main executable’s elf
> sections, which can break reasonable assumptions.  For example, musl’s
> dladdr fails to find symbols in the embedded linker, and gdb has
> trouble finding debug information from the linker.  Musl’s reuse of
> libc.so as the linker means that these problems apply to everything in
> libc.so, and also increases the size of every binary by including all
> of libc.so.
> 
> These problems with the embedded linker could be somewhat mitigated by
> splitting the dynamic linker out of libc.so when using the embedded
> linker.  That requires compiling the ldso sources against a statically
> linked libc.a, tweaking some of the initialization, and forwarding the
> dl* calls from libc.so to the separate linker.  The changes are
> relatively small, but result in a pretty big difference in musl’s
> internals with and without the embedded linker that may be hard to
> maintain.
> 

that breaks atomic update of the libc and introduces libc internal abi.
(i.e. bad for long term security and maintainability)

> The second solution we call “relinterp”.  It was originally designed
> by Ryan Prichard as a standalone trampoline that could be used with
> musl, glibc or bionic, but I’ve more tightly integrated it with musl
> in order to reuse CRTJMP for architecture portability and some of
> musl’s string functions to reduce the size of the code.  It uses a
> similar trampoline in Scrt1.o, but with a much larger implementation
> that reads DT_RUNPATH to construct a path to the dynamic linker that
> is relative to the executable.  It then maps the dynamic linker as the
> kernel would, modifies AT_BASE, AT_ENTRY and AT_PHDR, and jumps to the
> dynamic linker.
> 

i think this is a better approach.

i would not use Scrt1.o though, the same toolchain should be
usable for normal linking and relinterp linking, just use a
different name like Xcrt1.o.

> The current prototype of relinterp is tricky to compile, as it
> requires using -fvisibility=hidden and ld -r partial linking to build
> a Scrt1.o file that uses some of the src/string/*.c sources without
> any relocations, and then objcopy –keep-global-symbol to hide the
> string symbols.  It’s only useful if DT_RUNPATH contains $ORIGIN so
> that the dynamic linker can be distributed alongside the executable,
> so it is probably never going to be suitable for setuid binaries.
> 
> If relinterp were going to be included with musl I’d refactor it to
> reuse the __dls* bootstrapping from dynlink.c so that it can link
> against libc.a and not worry about avoiding any relocations.
> 

i would make Xcrt1.o self-contained and size optimized: it only
runs at start up, this is a different requirement from the -O3
build of normal string functions. and then there is no dependency
on libc internals (which may have various instrumentations that
does not work in Xcrt1.o).

> An alternative solution to these two would be to distribute statically
> linked binaries, which precludes the use of dlopen, or to wrap every
> executable in a shell script that runs the dynamic linker directly.
> 

i think it is possible to support static linking such that if dlopen
is linked then the entire libc gets linked into the main exe with
libc apis exported. then dlopen can work from an otherwise static exe.
(may not be easy to implement in practice though)

> Do either of these prototypes seem interesting enough to clean up and
> post as upstream patches, or should I keep them as a side project that
> I can bolt on to musl with minimal invasive changes?
> 
> Colin