Re: [musl] Supporting multilib LD_LIBRARY_PATHs

[Colin Cross <ccross@google.com>]

On Tue, Feb 7, 2023 at 4:49 PM Rich Felker <dalias@libc.org> wrote:
>
> On Tue, Feb 07, 2023 at 02:58:37PM -0800, Colin Cross wrote:
> > I'm hitting an issue where some test infrastructure is setting
> > LD_LIBRARY_PATH to a list that contains both 32-bit and 64-bit
> > libraries because it is unsure whether the code under test is going to
> > execute 32-bit or 64-bit processes or both.  When using musl the
> > dynamic loader takes the first library with a matching name and then
> > fails to load it if it is for the wrong elf class.
> >
> > The attached patch verifies the elf machine and class when searching
> > the path list, continuing the search if a valid elf header with an
> > incorrect machine or class is found.
>
> While it requires some consideration to ensure that this yields safe &
> consistent behavior, I think it at least admits that; I haven't
> checked the actual code, but conceptually, it should be equivalent to
> treating finding a mismatched-arch library as a conclusive result
> whose behavior is searching the remainder of the search path.
>
> I'm a little bit skeptical of the motivation though. In general, it's
> not safe to just set LD_LIBRARY_PATH and run programs that might
> invoke other programs, since the math might contain outdated or
> mismatched libraries relative to what those other programs might want.
> On a system with multiple libcs present, the libraries found could
> even be for the wrong one. Really, LD_LIBRARY_PATH should just be set
> for invoking a single program (or family of binaries) that ships with
> its own versions of libraries or when you're overriding certain
> libraries for it, etc. This is contrary to how the environment works,
> and one reason it's probably better to use ldso --library-path=...
> rather than LD_LIBRARY_PATH when overriding libraries for a particular
> program invocation.

We intend to distribute binaries built against musl using relinterp
(or Xcrt1 if that gets merged), which finds the dynamic loader
relative to the binary.  These will be used on end user systems that
normally use glibc.  The testing environment is taking a hermetic
bundle of musl and the test binaries built against musl, and are
intended to be hermetic so they shouldn't be executing anything that
would use a non-musl libc.  Normally the binaries use relative
DT_RUNPATH entries to find their libraries, but some older branches
don't have correct DT_RUNPATH entries for the layout of the tests vs.
their libraries, and so the test infrastructure uses LD_LIBRARY_PATH.
I'd like to remove the use of LD_LIBRARY_PATH at least in the cases
that have correct DT_RUNPATH entries, but this still seemed like a
reasonable enhancement to musl and will solve my immediate problems.

> If your goal is to change the default path that gets searched for all
> programs, the right place is /etc/ld-musl-$ARCH.path. It's already
> per-arch and non-conflicting with a potentially coexisting alternate
> libc.

We need to search for libraries relative to the binary, but if I'm
reading the code correctly relative paths in ../etc/ld-musl-$ARCH.path
would search relative to PWD.  The test infrastructure could write a
custom ../etc/ld-musl-$ARCH.path with absolute paths to wherever it
happened to install the bundle, but currently it is agnostic to the
details of the libc included in the bundle, and LD_LIBRARY_PATH works
for both glibc and musl (after this patch).

> Please don't take any of this as a rejection of the proposed change. I
> think it's probably okay, and probably an improvement on the usability
> of it. But I do think the apparent need suggests that .path files were
> likely a better solution to the problem at hand.
>
> Rich