mailing list of musl libc
 help / color / mirror / code / Atom feed
* [musl] Shared library loading
@ 2020-05-21 11:27 Alexander Scherbatiy
  2020-05-21 15:21 ` Markus Wichmann
  0 siblings, 1 reply; 14+ messages in thread
From: Alexander Scherbatiy @ 2020-05-21 11:27 UTC (permalink / raw)
  To: musl

Hello,

I use Alpine Linux 3.11.6 with musl libc (x86_64) version 1.1.24.

I have two shared libraries "a" and "b" each of which is placed in its 
own directory and lib "b" depends on "a".
First, I use dlopen to load lib "a" with "RTLD_NOW | RTLD_GLOBAL" flags.
Second, I load lib "b" with  flag "RTLD_LAZY".

The "b" library loading works on my Ubuntu 19.10 and fails on Alpine 
3.11.6 with message:
   dlopen failed: Error loading shared library liba.so: No such file or 
directory (needed by /root/load-lib-sample/bin/b/libb.so)
Should it work on Alpine with musl libc as well?


Below is the source code which reproduces the library loading issue:

--- include/a.h ---
void test_a();
--- include/b.h ---
void test_b();
--- src/a.c ---
#include "a.h"
#include <stdio.h>

void test_a() {
     printf("test a\n");
}
--- src/b.c ---
#include "b.h"
#include "a.h"
#include <stdio.h>

void test_b() {
     printf("test b\n");
     test_a();
}
--- src/main.c ---
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>

void *open_lib(const char* base_dir, const char* lib_dir, const char 
*lib_name,int flags) {
     char path_lib[256];
     sprintf(path_lib, "%s/%s/%s", base_dir, lib_dir, lib_name);
     printf("[load lib] %s\n", path_lib);

     void *handle = dlopen(path_lib, flags);

     if (!handle) {
         fprintf(stderr, "dlopen failed: %s\n", dlerror());
         exit(EXIT_FAILURE);
     }
     return handle;
}

void call_func(void *handle, const char *func_name) {
     printf("[call func] %s\n", func_name);

     void (*func_t)(void);
     *(void **) (&func_t) = dlsym(handle, func_name);

     char *error = dlerror();
     if (error != NULL)  {
         fprintf(stderr, "%s\n", error);
         exit(EXIT_FAILURE);
     }

     (*func_t)();
}

int main(int argc, char* argv[]) {
     if (argc < 2) {
         printf("provide base dir.\n");
         exit(-1);
     }

     void *handle_a = open_lib(argv[1], "a", "liba.so", RTLD_NOW | 
RTLD_GLOBAL);
     call_func(handle_a, "test_a");

     void *handle_b = open_lib(argv[1], "b", "libb.so", RTLD_LAZY);
     call_func(handle_b, "test_b");

     dlclose(handle_b);
     dlclose(handle_a);
}
--- build.sh ---
rm -rf ./bin
mkdir -p bin bin/a bin/b
DIR_BIN=$(pwd)/bin

gcc -g -shared src/a.c -Wl,-soname=liba.so -Iinclude -o bin/a/liba.so
gcc -g -shared src/b.c -Wl,-soname=libb.so -Iinclude -la -Lbin/a -o 
bin/b/libb.so
gcc -g src/main.c -Iinclude -o bin/main -ldl

./bin/main $DIR_BIN
--- --- ---

Output on Ubuntu 19.10
----------
./build.sh
[load lib] /home/user/load-lib-sample/bin/a/liba.so
[call func] test_a
test a
[load lib] /home/user/load-lib-sample/bin/b/libb.so
[call func] test_b
test b
test a
----------

Output on Alpine 3.11.6
----------
./build.sh
[load lib] /root/load-lib-sample/bin/a/liba.so
[call func] test_a
test a
[load lib] /root/load-lib-sample/bin/b/libb.so
dlopen failed: Error loading shared library liba.so: No such file or 
directory (needed by /root/load-lib-sample/bin/b/libb.so)
----------

Thanks,
Alexander.


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-21 11:27 [musl] Shared library loading Alexander Scherbatiy
@ 2020-05-21 15:21 ` Markus Wichmann
  2020-05-22 18:07   ` Alexander Scherbatiy
  0 siblings, 1 reply; 14+ messages in thread
From: Markus Wichmann @ 2020-05-21 15:21 UTC (permalink / raw)
  To: musl

On Thu, May 21, 2020 at 02:27:24PM +0300, Alexander Scherbatiy wrote:
> Hello,
>
> I use Alpine Linux 3.11.6 with musl libc (x86_64) version 1.1.24.
>
> I have two shared libraries "a" and "b" each of which is placed in its own
> directory and lib "b" depends on "a".
> First, I use dlopen to load lib "a" with "RTLD_NOW | RTLD_GLOBAL" flags.
> Second, I load lib "b" with  flag "RTLD_LAZY".
>
> The "b" library loading works on my Ubuntu 19.10 and fails on Alpine 3.11.6
> with message:
>   dlopen failed: Error loading shared library liba.so: No such file or
> directory (needed by /root/load-lib-sample/bin/b/libb.so)
> Should it work on Alpine with musl libc as well?

[...]

I shall leave the "Should it work" to others who know more about this
than me. But I will tackle the "Why doesn't it work". It appears however
the behavior is intentional (whether the consequences are as well
remains to be seen). Namely in line 1140f. of dynlink.c (revision
72658c65):


|/* Add a shortname only if name arg was not an explicit pathname. */
|if (pathname != name) p->shortname = strrchr(p->name, '/')+1;

This means, in your case, the DSO handle for liba does not get a short
name, so when libb is loaded, it is not detected that liba has already
been loaded (only the shortname is used for that purpose). And since
your files are in different directories, liba is also not found to be
loaded again. To work around this, you can set LD_LIBRARY_PATH to the
directory containing liba. In that case, when loading libb, the
dependency will be found with the environment search, and since liba was
already loaded, the loop in lines 1085ff. will find the handle and even
assign a shortname. The shortname is only used for checking of
duplicates.

Ciao,
Markus

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-21 15:21 ` Markus Wichmann
@ 2020-05-22 18:07   ` Alexander Scherbatiy
  2020-05-22 18:25     ` Rich Felker
  0 siblings, 1 reply; 14+ messages in thread
From: Alexander Scherbatiy @ 2020-05-22 18:07 UTC (permalink / raw)
  To: musl, Markus Wichmann

On 21.05.2020 18:21, Markus Wichmann wrote:

> On Thu, May 21, 2020 at 02:27:24PM +0300, Alexander Scherbatiy wrote:
>> Hello,
>>
>> I use Alpine Linux 3.11.6 with musl libc (x86_64) version 1.1.24.
>>
>> I have two shared libraries "a" and "b" each of which is placed in its own
>> directory and lib "b" depends on "a".
>> First, I use dlopen to load lib "a" with "RTLD_NOW | RTLD_GLOBAL" flags.
>> Second, I load lib "b" with  flag "RTLD_LAZY".
>>
>> The "b" library loading works on my Ubuntu 19.10 and fails on Alpine 3.11.6
>> with message:
>>    dlopen failed: Error loading shared library liba.so: No such file or
>> directory (needed by /root/load-lib-sample/bin/b/libb.so)
>> Should it work on Alpine with musl libc as well?
> [...]
>
> I shall leave the "Should it work" to others who know more about this
> than me. But I will tackle the "Why doesn't it work". It appears however
> the behavior is intentional (whether the consequences are as well
> remains to be seen). Namely in line 1140f. of dynlink.c (revision
> 72658c65):
>
>
> |/* Add a shortname only if name arg was not an explicit pathname. */
> |if (pathname != name) p->shortname = strrchr(p->name, '/')+1;

   It would be interesting to know which task this check is supposed to 
solve.

   Thanks,
   Alexander.

>
> This means, in your case, the DSO handle for liba does not get a short
> name, so when libb is loaded, it is not detected that liba has already
> been loaded (only the shortname is used for that purpose). And since
> your files are in different directories, liba is also not found to be
> loaded again. To work around this, you can set LD_LIBRARY_PATH to the
> directory containing liba. In that case, when loading libb, the
> dependency will be found with the environment search, and since liba was
> already loaded, the loop in lines 1085ff. will find the handle and even
> assign a shortname. The shortname is only used for checking of
> duplicates.
>
> Ciao,
> Markus

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-22 18:07   ` Alexander Scherbatiy
@ 2020-05-22 18:25     ` Rich Felker
  2020-05-23 14:12       ` Florian Weimer
  2020-05-25 17:26       ` Alexander Scherbatiy
  0 siblings, 2 replies; 14+ messages in thread
From: Rich Felker @ 2020-05-22 18:25 UTC (permalink / raw)
  To: Alexander Scherbatiy; +Cc: musl, Markus Wichmann

On Fri, May 22, 2020 at 09:07:20PM +0300, Alexander Scherbatiy wrote:
> On 21.05.2020 18:21, Markus Wichmann wrote:
> 
> >On Thu, May 21, 2020 at 02:27:24PM +0300, Alexander Scherbatiy wrote:
> >>Hello,
> >>
> >>I use Alpine Linux 3.11.6 with musl libc (x86_64) version 1.1.24.
> >>
> >>I have two shared libraries "a" and "b" each of which is placed in its own
> >>directory and lib "b" depends on "a".
> >>First, I use dlopen to load lib "a" with "RTLD_NOW | RTLD_GLOBAL" flags.
> >>Second, I load lib "b" with  flag "RTLD_LAZY".
> >>
> >>The "b" library loading works on my Ubuntu 19.10 and fails on Alpine 3.11.6
> >>with message:
> >>   dlopen failed: Error loading shared library liba.so: No such file or
> >>directory (needed by /root/load-lib-sample/bin/b/libb.so)
> >>Should it work on Alpine with musl libc as well?
> >[...]
> >
> >I shall leave the "Should it work" to others who know more about this
> >than me. But I will tackle the "Why doesn't it work". It appears however
> >the behavior is intentional (whether the consequences are as well
> >remains to be seen). Namely in line 1140f. of dynlink.c (revision
> >72658c65):
> >
> >
> >|/* Add a shortname only if name arg was not an explicit pathname. */
> >|if (pathname != name) p->shortname = strrchr(p->name, '/')+1;
> 
>   It would be interesting to know which task this check is supposed
> to solve.

The concept here is that non-pathname library names should be loaded
from the library path and not replaced by something (typically a
"module" or "plug-in") in a different, explicitly-loaded location that
happens to have a colliding base filename.

For example suppose your application loads modules from $libdir/myapp/
and has a module named "libfoo.so". Unbeknownst to you, there's also a
"libfoo.so" in the system paths, and some library you potentially load
indirectly (maybe the GPU driver for some video hardware you've never
heard of) depends on "libfoo.so".

If dlopen("$libdir/myapp/libfoo.so") had put "libfoo.so" in the
namespace such that it would satisfy future load requests for the name
"libfoo.so", the subsequent load would break due to getting the wrong
(unrelated) library.

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-22 18:25     ` Rich Felker
@ 2020-05-23 14:12       ` Florian Weimer
  2020-05-23 16:05         ` Rich Felker
  2020-05-25 17:26       ` Alexander Scherbatiy
  1 sibling, 1 reply; 14+ messages in thread
From: Florian Weimer @ 2020-05-23 14:12 UTC (permalink / raw)
  To: Rich Felker; +Cc: Alexander Scherbatiy, musl, Markus Wichmann

* Rich Felker:

>> >|/* Add a shortname only if name arg was not an explicit pathname. */
>> >|if (pathname != name) p->shortname = strrchr(p->name, '/')+1;
>> 
>>   It would be interesting to know which task this check is supposed
>> to solve.
>
> The concept here is that non-pathname library names should be loaded
> from the library path and not replaced by something (typically a
> "module" or "plug-in") in a different, explicitly-loaded location that
> happens to have a colliding base filename.
>
> For example suppose your application loads modules from $libdir/myapp/
> and has a module named "libfoo.so". Unbeknownst to you, there's also a
> "libfoo.so" in the system paths, and some library you potentially load
> indirectly (maybe the GPU driver for some video hardware you've never
> heard of) depends on "libfoo.so".
>
> If dlopen("$libdir/myapp/libfoo.so") had put "libfoo.so" in the
> namespace such that it would satisfy future load requests for the name
> "libfoo.so", the subsequent load would break due to getting the wrong
> (unrelated) library.

On the other hand, that breaks FFI implementatiosn which try to guess
the library name from some substring of it, and pass the absolute path
(as obtained from the file system) to dlopen.

(The root of the problem is that FFI users have not been taught to
load libraries by their soname (e.g., libsqlite3.so.0), but expect to
use strings such as "libsqlite3.so", "libsqlite3", or just "sqlite3".

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-23 14:12       ` Florian Weimer
@ 2020-05-23 16:05         ` Rich Felker
  0 siblings, 0 replies; 14+ messages in thread
From: Rich Felker @ 2020-05-23 16:05 UTC (permalink / raw)
  To: Florian Weimer; +Cc: Alexander Scherbatiy, musl, Markus Wichmann

On Sat, May 23, 2020 at 04:12:22PM +0200, Florian Weimer wrote:
> * Rich Felker:
> 
> >> >|/* Add a shortname only if name arg was not an explicit pathname. */
> >> >|if (pathname != name) p->shortname = strrchr(p->name, '/')+1;
> >> 
> >>   It would be interesting to know which task this check is supposed
> >> to solve.
> >
> > The concept here is that non-pathname library names should be loaded
> > from the library path and not replaced by something (typically a
> > "module" or "plug-in") in a different, explicitly-loaded location that
> > happens to have a colliding base filename.
> >
> > For example suppose your application loads modules from $libdir/myapp/
> > and has a module named "libfoo.so". Unbeknownst to you, there's also a
> > "libfoo.so" in the system paths, and some library you potentially load
> > indirectly (maybe the GPU driver for some video hardware you've never
> > heard of) depends on "libfoo.so".
> >
> > If dlopen("$libdir/myapp/libfoo.so") had put "libfoo.so" in the
> > namespace such that it would satisfy future load requests for the name
> > "libfoo.so", the subsequent load would break due to getting the wrong
> > (unrelated) library.
> 
> On the other hand, that breaks FFI implementatiosn which try to guess
> the library name from some substring of it, and pass the absolute path
> (as obtained from the file system) to dlopen.
> 
> (The root of the problem is that FFI users have not been taught to
> load libraries by their soname (e.g., libsqlite3.so.0), but expect to
> use strings such as "libsqlite3.so", "libsqlite3", or just "sqlite3".

I'm not sure what breakage you're thinking about. As long as the the
name resolves to the same file (dev/ino pair) as one already loaded,
the already-loaded one will be used. So it doesn't matter if you load
"libfoo.so.0" or "libfoo.so" as long as they're both links to the same
underlying file. The above-described distinction only happens when
there are actually different files, one in an absolute or relative
path provided by the caller (any string containing a '/'), and another
requested by dlopen or DT_NEEDED for search (no '/').

But yes Python is doing some really broken stuff in this area and
should be fixed.

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-22 18:25     ` Rich Felker
  2020-05-23 14:12       ` Florian Weimer
@ 2020-05-25 17:26       ` Alexander Scherbatiy
  2020-05-25 17:46         ` Rich Felker
  1 sibling, 1 reply; 14+ messages in thread
From: Alexander Scherbatiy @ 2020-05-25 17:26 UTC (permalink / raw)
  To: Rich Felker; +Cc: musl, Markus Wichmann

On 22.05.2020 21:25, Rich Felker wrote:

> The concept here is that non-pathname library names should be loaded
> from the library path and not replaced by something (typically a
> "module" or "plug-in") in a different, explicitly-loaded location that
> happens to have a colliding base filename.
>
> For example suppose your application loads modules from $libdir/myapp/
> and has a module named "libfoo.so". Unbeknownst to you, there's also a
> "libfoo.so" in the system paths, and some library you potentially load
> indirectly (maybe the GPU driver for some video hardware you've never
> heard of) depends on "libfoo.so".
>
> If dlopen("$libdir/myapp/libfoo.so") had put "libfoo.so" in the
> namespace such that it would satisfy future load requests for the name
> "libfoo.so", the subsequent load would break due to getting the wrong
> (unrelated) library.
   Is it correct that RTLD_GLOBAL flag is in some kind of obsolete in 
musl because calling
     dlopen("$libdir/myapp/libfoo.so", RTLD_NOW | RTLD_GLOBAL);
     dlopen("$libdir2/myapp2/libbar.so", RTLD_LAZY);

    does not allow libbar.so to find library $libdir/myapp/libfoo.so 
which it is depend on?

   What is the suggested way in musl to load $libdir/myapp/libfoo.so 
from a custom directory so libbar.so can find it?

Thanks,
Alexander.

> Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-25 17:26       ` Alexander Scherbatiy
@ 2020-05-25 17:46         ` Rich Felker
  2020-05-27 18:10           ` Dmitry Samersoff
  2020-06-04 19:37           ` Alexander Scherbatiy
  0 siblings, 2 replies; 14+ messages in thread
From: Rich Felker @ 2020-05-25 17:46 UTC (permalink / raw)
  To: Alexander Scherbatiy; +Cc: musl, Markus Wichmann

On Mon, May 25, 2020 at 08:26:51PM +0300, Alexander Scherbatiy wrote:
> On 22.05.2020 21:25, Rich Felker wrote:
> 
> >The concept here is that non-pathname library names should be loaded
> >from the library path and not replaced by something (typically a
> >"module" or "plug-in") in a different, explicitly-loaded location that
> >happens to have a colliding base filename.
> >
> >For example suppose your application loads modules from $libdir/myapp/
> >and has a module named "libfoo.so". Unbeknownst to you, there's also a
> >"libfoo.so" in the system paths, and some library you potentially load
> >indirectly (maybe the GPU driver for some video hardware you've never
> >heard of) depends on "libfoo.so".
> >
> >If dlopen("$libdir/myapp/libfoo.so") had put "libfoo.so" in the
> >namespace such that it would satisfy future load requests for the name
> >"libfoo.so", the subsequent load would break due to getting the wrong
> >(unrelated) library.
>   Is it correct that RTLD_GLOBAL flag is in some kind of obsolete in
> musl because calling
>     dlopen("$libdir/myapp/libfoo.so", RTLD_NOW | RTLD_GLOBAL);
>     dlopen("$libdir2/myapp2/libbar.so", RTLD_LAZY);
> 
>    does not allow libbar.so to find library $libdir/myapp/libfoo.so
> which it is depend on?

This is not what RTLD_GLOBAL means. RTLD_GLOBAL is a matter of whether
the library's symbols appear in the global symbol namespace to resolve
relocations in future-loaded libraries and dlsym with RTLD_DEFAULT. It
has nothing to do with whether the library is visible for satisfying
DT_NEEDED. The library is *always* visible for this purpose.

>   What is the suggested way in musl to load $libdir/myapp/libfoo.so
> from a custom directory so libbar.so can find it?

Probably putting the dir in your application's rpath (or in the rpath
of libbar). dlopen is not a tool for (effectively) adding additional
directories to the library search path.

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-25 17:46         ` Rich Felker
@ 2020-05-27 18:10           ` Dmitry Samersoff
  2020-05-28 18:53             ` Szabolcs Nagy
  2020-05-28 19:20             ` Jeffrey Walton
  2020-06-04 19:37           ` Alexander Scherbatiy
  1 sibling, 2 replies; 14+ messages in thread
From: Dmitry Samersoff @ 2020-05-27 18:10 UTC (permalink / raw)
  To: musl, Rich Felker, Alexander Scherbatiy; +Cc: Markus Wichmann

Hello Alexander,

> The "b" library loading works on my Ubuntu 19.10 and 
> fails on Alpine 3.11.6 with message: > dlopen failed: Error loading shared library liba.so: No such file or
> directory (needed by /root/load-lib-sample/bin/b/libb.so)
> Should it work on Alpine with musl libc as well?

1. You explicitly load library with a path (.../liba.so)

2. You are explicitly loading another library (.../libb.so)

3. Linker find liba.so in the appropriate section of libb.so and 
attempts to load it from syspath (LD_LIBRARY_PATH etc)

4. Linker doesn't find it. Musl return error on this step but glibc and 
BSD go further.

5. Linker compares short names of already loaded library and the 
required one

6. It matches, so Linker decides to resolve

I didn't find any specification that dictates one or other behavior, so 
it could not be considered as a bug.

-Dmitry

On 25.05.2020 20:46, Rich Felker wrote:
> On Mon, May 25, 2020 at 08:26:51PM +0300, Alexander Scherbatiy wrote:
>> On 22.05.2020 21:25, Rich Felker wrote:
>>
>>> The concept here is that non-pathname library names should be loaded
>> >from the library path and not replaced by something (typically a
>>> "module" or "plug-in") in a different, explicitly-loaded location that
>>> happens to have a colliding base filename.
>>>
>>> For example suppose your application loads modules from $libdir/myapp/
>>> and has a module named "libfoo.so". Unbeknownst to you, there's also a
>>> "libfoo.so" in the system paths, and some library you potentially load
>>> indirectly (maybe the GPU driver for some video hardware you've never
>>> heard of) depends on "libfoo.so".
>>>
>>> If dlopen("$libdir/myapp/libfoo.so") had put "libfoo.so" in the
>>> namespace such that it would satisfy future load requests for the name
>>> "libfoo.so", the subsequent load would break due to getting the wrong
>>> (unrelated) library.
>>    Is it correct that RTLD_GLOBAL flag is in some kind of obsolete in
>> musl because calling
>>      dlopen("$libdir/myapp/libfoo.so", RTLD_NOW | RTLD_GLOBAL);
>>      dlopen("$libdir2/myapp2/libbar.so", RTLD_LAZY);
>>
>>     does not allow libbar.so to find library $libdir/myapp/libfoo.so
>> which it is depend on?
> 
> This is not what RTLD_GLOBAL means. RTLD_GLOBAL is a matter of whether
> the library's symbols appear in the global symbol namespace to resolve
> relocations in future-loaded libraries and dlsym with RTLD_DEFAULT. It
> has nothing to do with whether the library is visible for satisfying
> DT_NEEDED. The library is *always* visible for this purpose.
> 
>>    What is the suggested way in musl to load $libdir/myapp/libfoo.so
>> from a custom directory so libbar.so can find it?
> 
> Probably putting the dir in your application's rpath (or in the rpath
> of libbar). dlopen is not a tool for (effectively) adding additional
> directories to the library search path.
> 
> Rich
> 


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-27 18:10           ` Dmitry Samersoff
@ 2020-05-28 18:53             ` Szabolcs Nagy
  2020-05-28 19:20             ` Jeffrey Walton
  1 sibling, 0 replies; 14+ messages in thread
From: Szabolcs Nagy @ 2020-05-28 18:53 UTC (permalink / raw)
  To: Dmitry Samersoff; +Cc: musl, Rich Felker, Alexander Scherbatiy, Markus Wichmann

* Dmitry Samersoff <dms@samersoff.net> [2020-05-27 21:10:18 +0300]:

> Hello Alexander,
> 
> > The "b" library loading works on my Ubuntu 19.10 and fails on Alpine
> > 3.11.6 with message: > dlopen failed: Error loading shared library
> > liba.so: No such file or
> > directory (needed by /root/load-lib-sample/bin/b/libb.so)
> > Should it work on Alpine with musl libc as well?
> 
> 1. You explicitly load library with a path (.../liba.so)
> 
> 2. You are explicitly loading another library (.../libb.so)
> 
> 3. Linker find liba.so in the appropriate section of libb.so and attempts to
> load it from syspath (LD_LIBRARY_PATH etc)
> 
> 4. Linker doesn't find it. Musl return error on this step but glibc and BSD
> go further.
> 
> 5. Linker compares short names of already loaded library and the required
> one

this step sounds like implementation specific heuristic
that is not mandated by any spec, so relying on it is
non-portable (and in this case can be dangerous: an
unrelated library may be used because of a name collision)

in principle a dynamic linker can continue even if it
didn't find a library with similar name at all, but that
would be dangerous too.

> 
> 6. It matches, so Linker decides to resolve
> 
> I didn't find any specification that dictates one or other behavior, so it
> could not be considered as a bug.

i think if there is no spec you should not rely on this
behaviour, i.e. it is a portability bug.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-27 18:10           ` Dmitry Samersoff
  2020-05-28 18:53             ` Szabolcs Nagy
@ 2020-05-28 19:20             ` Jeffrey Walton
  2020-05-28 19:29               ` Rich Felker
  1 sibling, 1 reply; 14+ messages in thread
From: Jeffrey Walton @ 2020-05-28 19:20 UTC (permalink / raw)
  To: musl

On Wed, May 27, 2020 at 2:10 PM Dmitry Samersoff <dms@samersoff.net> wrote:
>
> > The "b" library loading works on my Ubuntu 19.10 and
> > fails on Alpine 3.11.6 with message: > dlopen failed: Error loading shared library liba.so: No such file or
> > directory (needed by /root/load-lib-sample/bin/b/libb.so)
> > Should it work on Alpine with musl libc as well?
>
> 1. You explicitly load library with a path (.../liba.so)
>
> 2. You are explicitly loading another library (.../libb.so)
>
> 3. Linker find liba.so in the appropriate section of libb.so and
> attempts to load it from syspath (LD_LIBRARY_PATH etc)
>
> 4. Linker doesn't find it. Musl return error on this step but glibc and
> BSD go further.
>
> 5. Linker compares short names of already loaded library and the
> required one
>
> 6. It matches, so Linker decides to resolve
>
> I didn't find any specification that dictates one or other behavior, so
> it could not be considered as a bug.

I believe Drepper's DSO HowTo specifies the behavior for Linux.

Also see https://www.akkadia.org/drepper/dsohowto.pdf.

Jeff

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-28 19:20             ` Jeffrey Walton
@ 2020-05-28 19:29               ` Rich Felker
  0 siblings, 0 replies; 14+ messages in thread
From: Rich Felker @ 2020-05-28 19:29 UTC (permalink / raw)
  To: Jeffrey Walton; +Cc: musl

On Thu, May 28, 2020 at 03:20:46PM -0400, Jeffrey Walton wrote:
> On Wed, May 27, 2020 at 2:10 PM Dmitry Samersoff <dms@samersoff.net> wrote:
> >
> > > The "b" library loading works on my Ubuntu 19.10 and
> > > fails on Alpine 3.11.6 with message: > dlopen failed: Error loading shared library liba.so: No such file or
> > > directory (needed by /root/load-lib-sample/bin/b/libb.so)
> > > Should it work on Alpine with musl libc as well?
> >
> > 1. You explicitly load library with a path (.../liba.so)
> >
> > 2. You are explicitly loading another library (.../libb.so)
> >
> > 3. Linker find liba.so in the appropriate section of libb.so and
> > attempts to load it from syspath (LD_LIBRARY_PATH etc)
> >
> > 4. Linker doesn't find it. Musl return error on this step but glibc and
> > BSD go further.
> >
> > 5. Linker compares short names of already loaded library and the
> > required one
> >
> > 6. It matches, so Linker decides to resolve
> >
> > I didn't find any specification that dictates one or other behavior, so
> > it could not be considered as a bug.
> 
> I believe Drepper's DSO HowTo specifies the behavior for Linux.
> 
> Also see https://www.akkadia.org/drepper/dsohowto.pdf.

I'm not sure if the further search of already loaded libraries is
harmful or not, but Drepper's spec is not a standard we implement in
musl. It's certainly open to discussion but it's the type of behavior
that can easily have unforseen and bad consequences, and I would not
want to be hasty/sloppy about adopting it.

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-05-25 17:46         ` Rich Felker
  2020-05-27 18:10           ` Dmitry Samersoff
@ 2020-06-04 19:37           ` Alexander Scherbatiy
  2020-06-04 19:48             ` Rich Felker
  1 sibling, 1 reply; 14+ messages in thread
From: Alexander Scherbatiy @ 2020-06-04 19:37 UTC (permalink / raw)
  To: Rich Felker; +Cc: musl, Markus Wichmann

On 25.05.2020 20:46, Rich Felker wrote:

> On Mon, May 25, 2020 at 08:26:51PM +0300, Alexander Scherbatiy wrote:
>>    What is the suggested way in musl to load $libdir/myapp/libfoo.so
>> from a custom directory so libbar.so can find it?
> Probably putting the dir in your application's rpath (or in the rpath
> of libbar). dlopen is not a tool for (effectively) adding additional
> directories to the library search path.

OpenJDK can have several jvm variants (like server, client, minimal).
Each of variants contains its own version of libjvm.so lib.
   jdk/bin/java
   jdk/lib/server/libjvm.so
   jdk/lib/client/libjvm.so
   jdk/lib/minimal/libjvm.so

When java is executed it first loads libjvm.so for the requested variant
using dlopen() with RTLD_NOW | RTLD_GLOBAL flags [1] and then
loads libraries from jdk/lib dir with dlopen() and RTLD_LAZY flag.

To make it works with musl JDK implements workaround where
it puts libjvm.so into LD_LIBRARY_PATH [2] and re-executes itself.

Is there a better way to handle the case where an application
with several lib variants can load common libraries
without changing LD_LIBRARY_PATH?

[1] 
https://hg.openjdk.java.net/portola/portola/file/7ff60204a181/src/java.base/unix/native/libjli/java_md_solinux.c#l590
[2] 
https://hg.openjdk.java.net/portola/portola/file/7ff60204a181/src/java.base/unix/native/libjli/java_md_solinux.c#l244

Thanks,
Alexander.
> Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [musl] Shared library loading
  2020-06-04 19:37           ` Alexander Scherbatiy
@ 2020-06-04 19:48             ` Rich Felker
  0 siblings, 0 replies; 14+ messages in thread
From: Rich Felker @ 2020-06-04 19:48 UTC (permalink / raw)
  To: Alexander Scherbatiy; +Cc: musl, Markus Wichmann

On Thu, Jun 04, 2020 at 10:37:40PM +0300, Alexander Scherbatiy wrote:
> On 25.05.2020 20:46, Rich Felker wrote:
> 
> >On Mon, May 25, 2020 at 08:26:51PM +0300, Alexander Scherbatiy wrote:
> >>   What is the suggested way in musl to load $libdir/myapp/libfoo.so
> >>from a custom directory so libbar.so can find it?
> >Probably putting the dir in your application's rpath (or in the rpath
> >of libbar). dlopen is not a tool for (effectively) adding additional
> >directories to the library search path.
> 
> OpenJDK can have several jvm variants (like server, client, minimal).
> Each of variants contains its own version of libjvm.so lib.
>   jdk/bin/java
>   jdk/lib/server/libjvm.so
>   jdk/lib/client/libjvm.so
>   jdk/lib/minimal/libjvm.so
> 
> When java is executed it first loads libjvm.so for the requested variant
> using dlopen() with RTLD_NOW | RTLD_GLOBAL flags [1] and then
> loads libraries from jdk/lib dir with dlopen() and RTLD_LAZY flag.
> 
> To make it works with musl JDK implements workaround where
> it puts libjvm.so into LD_LIBRARY_PATH [2] and re-executes itself.
> 
> Is there a better way to handle the case where an application
> with several lib variants can load common libraries
> without changing LD_LIBRARY_PATH?
> 
> [1] https://hg.openjdk.java.net/portola/portola/file/7ff60204a181/src/java.base/unix/native/libjli/java_md_solinux.c#l590
> [2] https://hg.openjdk.java.net/portola/portola/file/7ff60204a181/src/java.base/unix/native/libjli/java_md_solinux.c#l244

One way to do it would be to make dummy libraries libjvm_server.so,
libjvm_client.so, and libjvm_minimal.so that contain no code or data
but just a DT_NEEDED reference to libjvm.so and an appropriate
DT_RUNPATH to find it.

Rich

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2020-06-04 19:49 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-21 11:27 [musl] Shared library loading Alexander Scherbatiy
2020-05-21 15:21 ` Markus Wichmann
2020-05-22 18:07   ` Alexander Scherbatiy
2020-05-22 18:25     ` Rich Felker
2020-05-23 14:12       ` Florian Weimer
2020-05-23 16:05         ` Rich Felker
2020-05-25 17:26       ` Alexander Scherbatiy
2020-05-25 17:46         ` Rich Felker
2020-05-27 18:10           ` Dmitry Samersoff
2020-05-28 18:53             ` Szabolcs Nagy
2020-05-28 19:20             ` Jeffrey Walton
2020-05-28 19:29               ` Rich Felker
2020-06-04 19:37           ` Alexander Scherbatiy
2020-06-04 19:48             ` Rich Felker

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/musl/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).