Re: [musl] Cannot dlopen() an already loaded shared library by its SONAME name

[Farid Zakaria <fmzakari@ucsc.edu>]

FWIW, I think we all agree that the discussion is involving libraries
that have a _soname_ and utilizing that as the cache key mechanism.

The other approach utilizing a symlink directory doesn't work with is
how I am trying to freeze dependencies using
https://github.com/fzakaria/shrinkwrap
The tool basically lifts all NEEDED dependencies from the whole
closure to the top level executable and requires them at absolute path
and does by trying to modify only
the executable ELF file.  Doing a symlink approach is a _lot more_
heavy handed to try and achieve the same effect.

On Wed, Jan 12, 2022 at 6:52 AM Harmen Stoppels <me@harmenstoppels.nl> wrote:
>
> > >
> > > 1.  Julia [1] splits binary dependencies into separate packages, so when
> > >     liba.so depends on libb.so, they live in a different dir, where
> > >     the absolute and relative paths are only known when the julia
> > >     interpreter has started, so neither rpaths or LD_LIBRARY_PATH can
> > >     be used.
> > >     So they dlopen libb.so, and then dlopen liba.so in that
> > >     order, and then assume liba.so does not have to locate libb.so
> > >     again, because its soname is already seens before.
> > >     The proposed workaround was: don't list libb.so in the
> > >     DT_NEEDED of liba.so (that is, if you're already doing the work of
> > >     the linker, you might as well not use the linker at all for locating
> > >     libs). However, being able to run executables shipped with julia
> > >     packages would still be nice (e.g. a subprocess with LD_LIBRARY_PATH
> > >     set properly)
> > >
> > > 2.  The Nix / Guix / Spack people are trying to reduce startup time of
> > >     executables with many shared libraries (as well as fixing library
> > >     paths once and for all to keep executables run deterministically).
> > >     In Guix there's a blog post where they call this the "stat storm" [2],
> > >     and they solve it in a glibc patch: using context dependent ld.so.cache,
> > >     that is, a reverse mapping soname => library path.
> > >     In Nix the proposal to fix the "stat storm" is to replace DT_NEEDED
> > >     in executables with absolute paths of all required libs (also
> > >     transient ones). This works fine, except on musl, where a dlopen by
> > >     soname will still do a search.
> >
> > This could be solved much better by making an application-specific
> > directory full of symlinks to the libraries it uses and putting that
> > directory as the first thing in the program binary's rpath.
>
> So the proposal is basically to replicate an application-specific
> ld.so.cache in the filesystem? Create a dir per ELF file, put its
> path in the RPATH, fill the dir with symlinks from soname => library.
>
> One problem with this is that $ORIGIN starts behaving differently
> compared to ld.so.cache. $ORIGIN is now relative to the directory
> of the symlink, not to the realpath of library. So if a library
> of a dependent package dlopen's a library in its own prefix by soname,
> relying on an rpath of say $ORIGIN/plugins, this will fail if the
> symlink dir is just a flat file list.
>
> So it means you'd effectively need to merge the prefixes, and this
> won't fly in Nix / Spack.
>
> Also it won't be a manageable solution for Julia, cause they require immutability
> of each prefix (and if they knew the relative path ahead of time, they
> wouldn't need this at all).
>
> > but having this happen on libraries without any SONAME is really an anti-feature.
>
> What do you mean? I think we're talking only about libraries that have a SONAME,
> do you mean dlopen-by-soname?
>
> So, do I understand correctly that loading a lib by path & putting its soname in
> a dict, so that future libs opened by soname can early exit is fine?
>
> But loading a lib by soname first, and then by path, and upon opening it
> happens to have a soname seen before, but is a different file (st_dev/st_ino),
> then it should continue with this lib, not early exit?