[musl] Dynamic linking of NEEDED with absolute path differs than that of glibc

[musl] Dynamic linking of NEEDED with absolute path differs than that of glibc

[Farid Zakaria]

Hello!

I am working, along with Tom <scogland1@llnl.gov> on a contribution to
patchelf[1] that takes all the NEEDED from downstream shared object
dependencies and lifts them to the top executable. I then change the
NEEDED in the top executable to be absolute paths. This seems to work
as expected in glibc.

Please use this example for demonstrations:

Here I have two shared objects libbar and libfoo, where foo depends on bar.
```
> patchelf --print-needed libbar.so
libc.so.6
> patchelf --print-soname libbar.so
libbar.so

> patchelf --print-needed libfoo.so
libbar.so
libc.so.6
```

I then add libbar and libfoo to another executable but reference them
in NEEDED with explicit paths.
```
> patchelf --print-needed tests/scratch/replace-add-needed/simple
/some-fixed-path/libbar.so
/some-fixed-path/libfoo.so
/lib/x86_64-linux-gnu/libc.so.6
```

glibc seems to happily link this and the executable runs.
```
ldd ./tests/scratch/replace-add-needed/simple
linux-vdso.so.1 (0x00007ffc98d98000)
/some-fixed-path/libbar.so (0x00007f269b6d3000)
/some-fixed-path/llibfoo.so (0x00007f269b6ce000)
/lib/x86_64-linux-gnu/libc.so.6 (0x00007f269b509000)
/lib64/ld-linux-x86-64.so.2 (0x00007f269c6e3000)
```

If I do the same however with musl it doesn't seem to link and
complains about not finding libbar.so
```
ldd ./tests/scratch/replace-add-needed/simple
/lib/ld-musl-x86_64.so.1 (0x7f04dd7cd000)
/some-fixed-path/libbar.so (0x7f04dc7bf000)
/some-fixed-path/libfoo.so (0x7f04dc7ba000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f04dd7cd000)
Error loading shared library libbar.so: No such file or directory
(needed by /some-fixed-path/libfoo.so)
```

Tom helped track this down due to the shortname assignment behavior[2]

This is clearly safer than what it was doing before that, but would
musl be open to either:
 1 - Using the contents of DT_SONAME if set, which matches the
behavior of glibc at least from observation, or
 2- Using a shortname based on the basename of the path even for
path-based loading?

It also seems that there are issues similar to this on the mailing
list recently as well [3] and a bit of a discussion from Rich
on the Julia Github repository that sounds similar [4].

Thank you!
Farid Zakaria

[1] https://github.com/NixOS/patchelf
[2] https://github.com/rofl0r/musl/commit/0420b874465db7544a9e1f320969b4920c9405d8
[3] https://www.openwall.com/lists/musl/2021/12/16/1
[4] https://github.com/JuliaLang/julia/issues/40556#issuecomment-929713727

Re: [musl] Dynamic linking of NEEDED with absolute path differs than that of glibc

[Markus Wichmann]

On Tue, Dec 21, 2021 at 10:56:33AM -0800, Farid Zakaria wrote:
> Hello!
>
> I am working, along with Tom <scogland1@llnl.gov> on a contribution to
> patchelf[1] that takes all the NEEDED from downstream shared object
> dependencies and lifts them to the top executable. I then change the
> NEEDED in the top executable to be absolute paths. This seems to work
> as expected in glibc.
>

May I ask why you copy the transitive dependencies? This also used to be
the default behavior of GNU ld, and it didn't make sense to me then,
either. I noticed it when I was using Gentoo. I had some library
providing some cryptographic service, but not actually implementing the
crypto itself; it could use at least two backends for that. Something
happened that broke one backend, so I recompiled the library to use the
other backend, but the executables now would not load, because they
contained a dependency on the library they never actually needed
directly. So I had to recompile the program. And I thought not having to
recompile the programs when you update the libraries was supposed to be
a strength of shared libraries.

>
> Tom helped track this down due to the shortname assignment behavior[2]
>

This has already been the subject of discussion years ago. The stated
reason (Rich gave) back then was that loading a library by explicit file
name is not the same as loading it by short name. Loading the library by
explicit file name can mean that you have a module with a private
version of a library, that is not meant to be globally accessible, and
if then later someone comes along to load a library by the short name,
they do not mean the same version, but rather, the system version of the
library.

Though I don't know if this applies to DT_NEEDED as much as to dlopen().

Ciao,
Markus

Re: [musl] Dynamic linking of NEEDED with absolute path differs than that of glibc

[Farid Zakaria]

On Tue, Dec 21, 2021 at 9:47 PM Markus Wichmann <nullplan@gmx.net> wrote:
>
> On Tue, Dec 21, 2021 at 10:56:33AM -0800, Farid Zakaria wrote:
> > Hello!
> >
> > I am working, along with Tom <scogland1@llnl.gov> on a contribution to
> > patchelf[1] that takes all the NEEDED from downstream shared object
> > dependencies and lifts them to the top executable. I then change the
> > NEEDED in the top executable to be absolute paths. This seems to work
> > as expected in glibc.
> >
>
> May I ask why you copy the transitive dependencies? This also used to be
> the default behavior of GNU ld, and it didn't make sense to me then,
> either. I noticed it when I was using Gentoo. I had some library
> providing some cryptographic service, but not actually implementing the
> crypto itself; it could use at least two backends for that. Something
> happened that broke one backend, so I recompiled the library to use the
> other backend, but the executables now would not load, because they
> contained a dependency on the library they never actually needed
> directly. So I had to recompile the program. And I thought not having to
> recompile the programs when you update the libraries was supposed to be
> a strength of shared libraries.

Tom and I primarily contribute (Spack & NixOS) to tooling for setups
where having explicit paths is a desired effect.
These tools tend to have very large RUNPATHs that are searched for
every NEEDED, making the location explicit in the NEEDED is one
desirable outcome to avoid this search.

Tom answered this off-list which I will include here as well:

We’re looking at this partly as a research project but also partly
because we have recurring issues with vendor-provided packages leaving
us with incomplete, incorrect, or mixed library search setups.  Things
like an internally consistent set of runpaths for everything from the
vendor but no runpath for specific-version dependencies they require,
or mixing RPATH on some libraries with RUNPATH on others, or injecting
LD_LIBRARY_PATH into modules or wrappers, sometimes several of these
at the same time.

The TLDR is we’re experimenting with ways we can produce a binary that
will load the entire tree of dependencies, including transitive ones,
exactly as specified regardless of the state of the paths embedded in
dependencies we may not control.  As a first step in that direction,
we’re getting to having it load everything the way it would have been
loaded if the application were run in the exact same context as the
linker.

Leveraging the caching of needed entries to allow us to directly load
the entire graph in the actual application works surprisingly well,
but it’s predicated on being able to ensure that a search for
“libbar.so” from a dependency finds our loaded libbar.so, currently
loaded by path, rather than searching again, since the search may fail
or return with an incompatible library.


> >
> > Tom helped track this down due to the shortname assignment behavior[2]
> >
>
> This has already been the subject of discussion years ago. The stated
> reason (Rich gave) back then was that loading a library by explicit file
> name is not the same as loading it by short name. Loading the library by
> explicit file name can mean that you have a module with a private
> version of a library, that is not meant to be globally accessible, and
> if then later someone comes along to load a library by the short name,
> they do not mean the same version, but rather, the system version of the
> library.
>
> Though I don't know if this applies to DT_NEEDED as much as to dlopen().
>
> Ciao,
> Markus

Thank you for the added context on the rationale thus far. I'm
wondering if there's a way forward to achieve our desired effect with
or without a patch.
We've audited a few linkers and surprisingly this aspect is not well
documented and varies across many loaders and across OS (i.e.
Darwin/Windows).

Re: [musl] Dynamic linking of NEEDED with absolute path differs than that of glibc

[Adhemerval Zanella]

On 21/12/2021 15:56, Farid Zakaria wrote:
> ```
> 
> glibc seems to happily link this and the executable runs.
> ```
> ldd ./tests/scratch/replace-add-needed/simple
> linux-vdso.so.1 (0x00007ffc98d98000)
> /some-fixed-path/libbar.so (0x00007f269b6d3000)
> /some-fixed-path/llibfoo.so (0x00007f269b6ce000)
> /lib/x86_64-linux-gnu/libc.so.6 (0x00007f269b509000)
> /lib64/ld-linux-x86-64.so.2 (0x00007f269c6e3000)
> ```
> 

Not sure if this is what you are doing here, but keep in mind that
mixing glibc dynamic loader and libc with different versions is not
supported and can break any time.

Re: [musl] Dynamic linking of NEEDED with absolute path differs than that of glibc

[Tom Scogland]

Certainly, we have no intention of mixing different versions of loader and libc or different libc types.  In fact it’s mainly about making sure we avoid getting incorrectly mixed versions of other libraries due to loading issues.

On 23 Dec 2021, at 3:46, Adhemerval Zanella wrote:

> On 21/12/2021 15:56, Farid Zakaria wrote:
>> ```
>>
>> glibc seems to happily link this and the executable runs.
>> ```
>> ldd ./tests/scratch/replace-add-needed/simple
>> linux-vdso.so.1 (0x00007ffc98d98000)
>> /some-fixed-path/libbar.so (0x00007f269b6d3000)
>> /some-fixed-path/llibfoo.so (0x00007f269b6ce000)
>> /lib/x86_64-linux-gnu/libc.so.6 (0x00007f269b509000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007f269c6e3000)
>> ```
>>
>
> Not sure if this is what you are doing here, but keep in mind that
> mixing glibc dynamic loader and libc with different versions is not
> supported and can break any time.