Re: [musl] Supporting multilib LD_LIBRARY_PATHs

[Rich Felker <dalias@libc.org>]

On Tue, Feb 07, 2023 at 09:28:10PM -0800, Colin Cross wrote:
> On Tue, Feb 7, 2023 at 4:49 PM Rich Felker <dalias@libc.org> wrote:
> >
> > On Tue, Feb 07, 2023 at 02:58:37PM -0800, Colin Cross wrote:
> > > I'm hitting an issue where some test infrastructure is setting
> > > LD_LIBRARY_PATH to a list that contains both 32-bit and 64-bit
> > > libraries because it is unsure whether the code under test is going to
> > > execute 32-bit or 64-bit processes or both.  When using musl the
> > > dynamic loader takes the first library with a matching name and then
> > > fails to load it if it is for the wrong elf class.
> > >
> > > The attached patch verifies the elf machine and class when searching
> > > the path list, continuing the search if a valid elf header with an
> > > incorrect machine or class is found.
> >
> > While it requires some consideration to ensure that this yields safe &
> > consistent behavior, I think it at least admits that; I haven't
> > checked the actual code, but conceptually, it should be equivalent to
> > treating finding a mismatched-arch library as a conclusive result
> > whose behavior is searching the remainder of the search path.
> >
> > I'm a little bit skeptical of the motivation though. In general, it's
> > not safe to just set LD_LIBRARY_PATH and run programs that might
> > invoke other programs, since the math might contain outdated or
> > mismatched libraries relative to what those other programs might want.
> > On a system with multiple libcs present, the libraries found could
> > even be for the wrong one. Really, LD_LIBRARY_PATH should just be set
> > for invoking a single program (or family of binaries) that ships with
> > its own versions of libraries or when you're overriding certain
> > libraries for it, etc. This is contrary to how the environment works,
> > and one reason it's probably better to use ldso --library-path=...
> > rather than LD_LIBRARY_PATH when overriding libraries for a particular
> > program invocation.
> 
> We intend to distribute binaries built against musl using relinterp
> (or Xcrt1 if that gets merged), which finds the dynamic loader
> relative to the binary.  These will be used on end user systems that
> normally use glibc.  The testing environment is taking a hermetic
> bundle of musl and the test binaries built against musl, and are
> intended to be hermetic so they shouldn't be executing anything that
> would use a non-musl libc.  Normally the binaries use relative
> DT_RUNPATH entries to find their libraries, but some older branches
> don't have correct DT_RUNPATH entries for the layout of the tests vs.
> their libraries, and so the test infrastructure uses LD_LIBRARY_PATH.
> I'd like to remove the use of LD_LIBRARY_PATH at least in the cases
> that have correct DT_RUNPATH entries, but this still seemed like a
> reasonable enhancement to musl and will solve my immediate problems.

OK, this makes a lot of sense. Thanks for explaining!

> > If your goal is to change the default path that gets searched for all
> > programs, the right place is /etc/ld-musl-$ARCH.path. It's already
> > per-arch and non-conflicting with a potentially coexisting alternate
> > libc.
> 
> We need to search for libraries relative to the binary, but if I'm
> reading the code correctly relative paths in ../etc/ld-musl-$ARCH.path
> would search relative to PWD.  The test infrastructure could write a
> custom ../etc/ld-musl-$ARCH.path with absolute paths to wherever it
> happened to install the bundle, but currently it is agnostic to the
> details of the libc included in the bundle, and LD_LIBRARY_PATH works
> for both glibc and musl (after this patch).

OK, that seems to be an important oversight: the inability to have
$ORIGIN relative (or, ideally, ldso-relative) library paths in the
ld-musl-*.path files. Maybe rpath kinda reduces the need for that,
since you can just put the rpaths in the application binaries, but it
still seems like it'd be more elegant (and more flexible) to have it
in the path files.

I think it's also worth asking if there should be different default
library path (in the absence of a path file) when ldso is not in /lib.
Possible reasonable-sounding behaviors would be no path at all
(everything fails without an explicit path file or LD_LIBRARY_PATH or
rpath) or having a default path that's just the same directory as ldso
resides in. (In some sense, the latter is what we probably should have
done all along even for installation in /lib, but having /usr/lib and
/usr/local/lib also included was a nice convenience... :/)

Rich