musl ldd: swt build: Error relocating / symbol not found

musl ldd: swt build: Error relocating / symbol not found

[Andrei Pozolotin]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

    Hello.

    I have a curious problem:

    1) the following build system:
    https://github.com/random-alpiner/repository/blob/master/bugs/01/APKBUILD
    https://github.com/random-alpiner/repository/blob/master/bugs/01/build.sh
    https://github.com/random-alpiner/repository/blob/master/bugs/01/make_linux.mak

    2) produces a library:
    https://github.com/random-alpiner/repository/blob/master/alpine/x86_64/swt-4.5.2-r1.apk

    3) which, when loaded by target application (java/eclipse), produces
    an error:
    https://github.com/random-alpiner/repository/blob/master/bugs/01/1-app.log

    4) and yet, that error should not happen, according to musl ldd:
    https://github.com/random-alpiner/repository/blob/master/bugs/01/2-ldd.log

    5) and also confirmed by readelf:
    https://github.com/random-alpiner/repository/blob/master/bugs/01/3-readelf.log

    Any pointers on how to resolve this seeming contradiction are much
    appreciated.

    Finally, someone on irc#musl suggested that musl ldd provides "-d"
    and "-r" options
    by default out of the box:
    http://man7.org/linux/man-pages/man1/ldd.1.html. Is that so?

    Thank you,

    Andrei


[-- Attachment #2: Type: text/html, Size: 2676 bytes --]

Re: musl ldd: swt build: Error relocating / symbol not found

[Rich Felker]

On Thu, Jun 23, 2016 at 03:35:02AM +0000, Andrei Pozolotin wrote:
>     Hello.
> 
>     I have a curious problem:
> 
>     1) the following build system:
>     https://github.com/random-alpiner/repository/blob/master/bugs/01/APKBUILD
>     https://github.com/random-alpiner/repository/blob/master/bugs/01/build.sh
>     https://github.com/random-alpiner/repository/blob/master/bugs/01/make_linux.mak
> 
>     2) produces a library:
>     https://github.com/random-alpiner/repository/blob/master/alpine/x86_64/swt-4.5.2-r1.apk
> 
>     3) which, when loaded by target application (java/eclipse), produces
>     an error:
>     https://github.com/random-alpiner/repository/blob/master/bugs/01/1-app.log
> 
>     4) and yet, that error should not happen, according to musl ldd:
>     https://github.com/random-alpiner/repository/blob/master/bugs/01/2-ldd.log
> 
>     5) and also confirmed by readelf:
>     https://github.com/random-alpiner/repository/blob/master/bugs/01/3-readelf.log
> 
>     Any pointers on how to resolve this seeming contradiction are much
>     appreciated.

My first guess would be that something else in the application
(eclipse) has already caused an older/stale version of libgobject to
be loaded, so that the version containing the symbol definition does
not get loaded. You could confirm this by running strace on the
program and checking what library files it loads/maps.

>     Finally, someone on irc#musl suggested that musl ldd provides "-d"
>     and "-r" options
>     by default out of the box:
>     http://man7.org/linux/man-pages/man1/ldd.1.html. Is that so?

This is roughly correct. Symbol resolution is always attempted and
errors reported.

Rich

Re: musl ldd: swt build: Error relocating / symbol not found

[Andrei Pozolotin]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

Rich:

On 06/23/2016 04:24 AM, Rich Felker wrote:
> On Thu, Jun 23, 2016 at 03:35:02AM +0000, Andrei Pozolotin wrote:
>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/1-app.log
>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/2-ldd.log
>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/3-readelf.log
>>
>> My first guess would be that something else in the application
>> (eclipse) has already caused an older/stale version of libgobject to
>> be loaded, so that the version containing the symbol definition does
>> not get loaded. You could confirm this by running strace on the
>> program and checking what library files it loads/maps.
Thank you for the tip.

1) the trace shows:
https://github.com/random-alpiner/repository/blob/master/bugs/01/4-strace.log

2) that the proper /usr/lib/libgobject-2.0.so.0 is in fact loaded before
/usr/lib/libswt-atk-gtk-4530.so

3) however, the loading starts from
/home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-gtk-4530.so,

which is an eclipse default library bundled with the eclipse
application, and which is compiled against foreign libc.6.so:

lddtree /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-gtk-4530.so
libswt-gtk-4530.so => /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-gtk-4530.so (interpreter => none)
    libc.so.6 => not found

4) the library  loading logic of eclipse is: first try to load bundled
libraries from private paths,
and if that fails, then try to load these libraries from the public
paths, such as "/usr/lib/*", etc.

5) I do provide replacement libswt-* libraries built against musl on the
public path "/usr/lib/*",
and the assumption was that when private path library loading fails,
then the library will be loaded from the public path, which "sort of
almost works",
so:
* was that a wrong assumption? 
* will in fact musl ld.so reject libraries linked to the libc.so.6?
* if not, is there a way to tell musl ld.so to blacklist private path
libraries?


[-- Attachment #2: Type: text/html, Size: 4976 bytes --]

Re: musl ldd: swt build: Error relocating / symbol not found

[Rich Felker]

On Thu, Jun 23, 2016 at 03:39:46PM +0000, Andrei Pozolotin wrote:
> Rich:
> 
> On 06/23/2016 04:24 AM, Rich Felker wrote:
> > On Thu, Jun 23, 2016 at 03:35:02AM +0000, Andrei Pozolotin wrote:
> >>     https://github.com/random-alpiner/repository/blob/master/bugs/01/1-app.log
> >>     https://github.com/random-alpiner/repository/blob/master/bugs/01/2-ldd.log
> >>     https://github.com/random-alpiner/repository/blob/master/bugs/01/3-readelf.log
> >>
> >> My first guess would be that something else in the application
> >> (eclipse) has already caused an older/stale version of libgobject to
> >> be loaded, so that the version containing the symbol definition does
> >> not get loaded. You could confirm this by running strace on the
> >> program and checking what library files it loads/maps.
> Thank you for the tip.
> 
> 1) the trace shows:
> https://github.com/random-alpiner/repository/blob/master/bugs/01/4-strace.log
> 
> 2) that the proper /usr/lib/libgobject-2.0.so.0 is in fact loaded before
> /usr/lib/libswt-atk-gtk-4530.so
> 
> 3) however, the loading starts from
> /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-gtk-4530.so,
> 
> which is an eclipse default library bundled with the eclipse
> application, and which is compiled against foreign libc.6.so:
> 
> lddtree /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-gtk-4530.so
> libswt-gtk-4530.so => /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-gtk-4530.so (interpreter => none)
>     libc.so.6 => not found

OK so it looks like you're trying to run foreign glibc binaries with
musl. That might or might not work, and in this case it seems to be
not working. You really should be using a build of eclipse and all its
libraries against musl rather than putting glibc-linked binaries on
the system.

> 4) the library  loading logic of eclipse is: first try to load bundled
> libraries from private paths,
> and if that fails, then try to load these libraries from the public
> paths, such as "/usr/lib/*", etc.
> 
> 5) I do provide replacement libswt-* libraries built against musl on the
> public path "/usr/lib/*",
> and the assumption was that when private path library loading fails,
> then the library will be loaded from the public path, which "sort of
> almost works",
> so:
> * was that a wrong assumption? 
> * will in fact musl ld.so reject libraries linked to the libc.so.6?

No, it will use them.

> * if not, is there a way to tell musl ld.so to blacklist private path
> libraries?

No, but you could just rm (or mv) them. But I suspect you'll still run
into other problems. Support for using glibc-linked programs/libraries
is inherently error-prone and really only intended for use in
situations where you can't get a properly-built binary (e.g. Flash
plugin).

Rich

Re: musl ldd: swt build: Error relocating / symbol not found

[Andrei Pozolotin]

Rich:

On 06/23/2016 05:10 PM, Rich Felker wrote:
>>> On Thu, Jun 23, 2016 at 03:35:02AM +0000, Andrei Pozolotin wrote:
>>>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/1-app.log
>>>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/2-ldd.log
>>>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/3-readelf.log
>>>>     https://github.com/random-alpiner/repository/blob/master/bugs/01/4-strace.log
> OK so it looks like you're trying to run foreign glibc binaries with
> musl. That might or might not work, and in this case it seems to be
no, the intent not to use foreign glibc libraries at all.
but they do sneak in :-)
> not working. You really should be using a build of eclipse and all its
> libraries against musl rather than putting glibc-linked binaries on
> the system.
full eclipse build is large and heavy,
and "override native swt.so" approach worked for us in the past
>
>> * is there a way to tell musl ld.so to blacklist private path libraries?
> No, but you could just rm (or mv) them. But I suspect you'll still run
> into other problems. Support for using glibc-linked programs/libraries
> is inherently error-prone and really only intended for use in
> situations where you can't get a properly-built binary (e.g. Flash
> plugin).
>
> Rich
1) the "other" problems are fine, when we get to them :-)
we do intend to override all default eclipse glibc-linked libraries,
via approach described in this conversation, so that would not be an issue

2) we found a way to switch eclipse library loading logic to the following:
a) try to load public path (i.e.  provided musl libc dependent)
libraries - first
b) then try to load private path (i.t. bundled foreign libc dependent)
libraries - second

3) after the change in the loading logic, the trace now shows that:
https://github.com/random-alpiner/repository/blob/master/bugs/01/5-strace.log

libswt-* ---> ... ---> libgobject [g_type_check_class_cast] 

loading chain is now 100% pure musl libc only

4) nonetheless, we still have the "original contradiction problem":

a) the application error is bubbling up form the musl ld.so: "Error
relocating / symbol not found":

java.lang.UnsatisfiedLinkError: Could not load SWT library. Reasons: 
    /usr/lib/libswt-atk-gtk-4530.so: Error relocating /usr/lib/libswt-atk-gtk-4530.so: g_type_check_class_cast: symbol not found
    /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-atk-gtk-4530.so: Error relocating /home/work/space/none-4.5.0/conf/org.eclipse.osgi/748/0/.cp/libswt-atk-gtk-4530.so: g_type_interface_peek: symbol not found
    /usr/lib/libswt-atk-gtk-4530.so: Error relocating /usr/lib/libswt-atk-gtk-4530.so: g_type_check_class_cast: symbol not found


b) while at the same time musl ldd reporting that library dependency
tree is resolved with no error:

lddtree /usr/lib/libswt-atk-gtk-4530.so
libswt-atk-gtk-4530.so => /usr/lib/libswt-atk-gtk-4530.so (interpreter => none)
    libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0
        libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0
            libffi.so.6 => /usr/lib/libffi.so.6
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0
            libpcre.so.1 => /usr/lib/libpcre.so.1
        libintl.so.8 => /usr/lib/libintl.so.8
    libc.musl-x86_64.so.1 => /lib/libc.musl-x86_64.so.1


5) so it seems we are indeed dealing with genuine musl ld.so issue?

Re: musl ldd: swt build: Error relocating / symbol not found

[Szabolcs Nagy]

* Andrei Pozolotin <andrei.pozolotin@gmail.com> [2016-06-23 19:42:44 +0000]:
> b) while at the same time musl ldd reporting that library dependency
> tree is resolved with no error:
> 
> lddtree /usr/lib/libswt-atk-gtk-4530.so

that's not musl's ldd, but scanelf from pax-utils

when debugging such a complicated setup the output
of tools that may use subtly different library paths
and symbol resolution logic is not very helpful.

ldd /usr/lib/libswt-gtk-4530.so
ldd /usr/lib/libswt-atk-gtk-4530.so

would be more interesting..

but even then we don't know what's going on
(if libswt-gtk-4530.so is dlopened with RTLD_LOCAL
then its libgobject dependency might not be visible
to libswt-atk-gtk-4530)

Re: musl ldd: swt build: Error relocating / symbol not found

[Andrei Pozolotin]

[-- Attachment #1: Type: text/plain, Size: 3046 bytes --]

Szabolcs:

On 06/23/2016 11:15 PM, Szabolcs Nagy wrote:
> * Andrei Pozolotin <andrei.pozolotin@gmail.com> [2016-06-23 19:42:44 +0000]:
>> b) while at the same time musl ldd reporting that library dependency
>> tree is resolved with no error:
>>
>> lddtree /usr/lib/libswt-atk-gtk-4530.so
> that's not musl's ldd, but scanelf from pax-utils
thank you for pointing out.
> when debugging such a complicated setup the output
> of tools that may use subtly different library paths
> and symbol resolution logic is not very helpful.
ok, got it.
> ldd /usr/lib/libswt-gtk-4530.so
ldd /usr/lib/libswt-gtk-4530.so
        ldd (0x55e333e6c000)
        libc.musl-x86_64.so.1 => ldd (0x55e333e6c000)
> ldd /usr/lib/libswt-atk-gtk-4530.so
ldd /usr/lib/libswt-atk-gtk-4530.so
        ldd (0x55edc6edc000)
        libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0 (0x7fc763298000)
        libc.musl-x86_64.so.1 => ldd (0x55edc6edc000)
        libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x7fc763058000)
        libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x7fc762d6d000)
        libintl.so.8 => /usr/lib/libintl.so.8 (0x7fc762b5f000)
        libffi.so.6 => /usr/lib/libffi.so.6 (0x7fc762957000)
        libpcre.so.1 => /usr/lib/libpcre.so.1 (0x7fc7626fe000)
> would be more interesting..
>
> but even then we don't know what's going on
> (if libswt-gtk-4530.so is dlopened with RTLD_LOCAL
> then its libgobject dependency might not be visible
> to libswt-atk-gtk-4530)
OK. here is the story:

* java native interface: NativeLibrary.load()
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/74e5fc94c77b/src/share/classes/java/lang/ClassLoader.java#l1726

* java JNI implementation:
Java_java_lang_ClassLoader_00024NativeLibrary_load
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/74e5fc94c77b/src/share/native/java/lang/ClassLoader.c#l369

* libjvm.so entry point: os::dll_load
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l3959

* libjvm.so linux implementation
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/share/vm/runtime/os.hpp#l564
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1773
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1767
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1997
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1988

* and finally: it says: dlopen RTLD_LAZY:
http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1988
void * result = ::dlopen(filename, RTLD_LAZY);

http://linux.die.net/man/3/dlopen
RTLD_LAZY: Perform lazy binding. Only resolve symbols as the code that
references them is executed.
If the symbol is never referenced, then it is never resolved.
(Lazy binding is only performed for function references;
references to variables are always immediately bound when the library is
loaded.)

RTLD_LAZY is good, right? :-)


[-- Attachment #2: Type: text/html, Size: 6700 bytes --]

Re: musl ldd: swt build: Error relocating / symbol not found

[Rich Felker]

On Fri, Jun 24, 2016 at 04:23:55PM +0000, Andrei Pozolotin wrote:
> Szabolcs:
> 
> On 06/23/2016 11:15 PM, Szabolcs Nagy wrote:
> > * Andrei Pozolotin <andrei.pozolotin@gmail.com> [2016-06-23 19:42:44 +0000]:
> >> b) while at the same time musl ldd reporting that library dependency
> >> tree is resolved with no error:
> >>
> >> lddtree /usr/lib/libswt-atk-gtk-4530.so
> > that's not musl's ldd, but scanelf from pax-utils
> thank you for pointing out.
> > when debugging such a complicated setup the output
> > of tools that may use subtly different library paths
> > and symbol resolution logic is not very helpful.
> ok, got it.
> > ldd /usr/lib/libswt-gtk-4530.so
> ldd /usr/lib/libswt-gtk-4530.so
>         ldd (0x55e333e6c000)
>         libc.musl-x86_64.so.1 => ldd (0x55e333e6c000)
> > ldd /usr/lib/libswt-atk-gtk-4530.so
> ldd /usr/lib/libswt-atk-gtk-4530.so
>         ldd (0x55edc6edc000)
>         libatk-1.0.so.0 => /usr/lib/libatk-1.0.so.0 (0x7fc763298000)
>         libc.musl-x86_64.so.1 => ldd (0x55edc6edc000)
>         libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x7fc763058000)
>         libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x7fc762d6d000)
>         libintl.so.8 => /usr/lib/libintl.so.8 (0x7fc762b5f000)
>         libffi.so.6 => /usr/lib/libffi.so.6 (0x7fc762957000)
>         libpcre.so.1 => /usr/lib/libpcre.so.1 (0x7fc7626fe000)
> > would be more interesting..
> >
> > but even then we don't know what's going on
> > (if libswt-gtk-4530.so is dlopened with RTLD_LOCAL
> > then its libgobject dependency might not be visible
> > to libswt-atk-gtk-4530)
> OK. here is the story:
> 
> * java native interface: NativeLibrary.load()
> http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/74e5fc94c77b/src/share/classes/java/lang/ClassLoader.java#l1726
> 
> * java JNI implementation:
> Java_java_lang_ClassLoader_00024NativeLibrary_load
> http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/74e5fc94c77b/src/share/native/java/lang/ClassLoader.c#l369
> 
> * libjvm.so entry point: os::dll_load
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/tip/src/share/vm/prims/jvm.cpp#l3959
> 
> * libjvm.so linux implementation
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/share/vm/runtime/os.hpp#l564
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1773
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1767
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1997
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1988
> 
> * and finally: it says: dlopen RTLD_LAZY:
> http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/file/4529ee76d3f9/src/os/linux/vm/os_linux.cpp#l1988
> void * result = ::dlopen(filename, RTLD_LAZY);
> 
> http://linux.die.net/man/3/dlopen
> RTLD_LAZY: Perform lazy binding. Only resolve symbols as the code that
> references them is executed.
> If the symbol is never referenced, then it is never resolved.
> (Lazy binding is only performed for function references;
> references to variables are always immediately bound when the library is
> loaded.)
> 
> RTLD_LAZY is good, right? :-)

OK, this is likely the root of the problem: invalid code assuming that
it can load libraries with undefined symbols as long as it doesn't try
to use those code paths.

The man page you linked to is rather poor-quality. When symbol binding
takes place with RTLD_LAZY is actually implementation-defined and can
be anywhere between the time of dlopen and the time of use. The flag
should be treated only as a hint for allowing performance
optimizations, not as something that gives the caller permission to do
erroneous things.

Aside from formal correctness, there are multiple reasons for this.
It's architecture- and linktime-option-dependent whether late binding
is even possible at all, and musl purposefully does not implement lazy
binding because it's a huge surface for bugs (which you can see by
looking at glibc's history of bugs caused by lazy binding).

There's one other well-known piece of software, x.org, abusing
RTLD_LAZY in the same way, and we have discussed possible workarounds
before. It would be possible to accept relocations with undefined
symbol references at dlopen time by storing a list of them, and rather
than lazily processing them at call time, re-process them after each
additional dlopen. This would allow broken programs to work without
introducing the bug surface that actual lazy-binding introduces.
However it's a fairly big task to add, and it would be much nicer just
to get the buggy programs fixed (there are already reasonable
workarounds for x.org).

Rich

Re: musl ldd: swt build: Error relocating / symbol not found

[Alexander Monakov]

On Fri, 24 Jun 2016, Rich Felker wrote:
> OK, this is likely the root of the problem: invalid code assuming that
> it can load libraries with undefined symbols as long as it doesn't try
> to use those code paths.

No, this does not explain the issue: as the first message in thread shows, the
symbol is defined in libgobject, and if it wasn't, ldd would detect that. So
we still don't have the root cause.

Alexander

Re: musl ldd: swt build: Error relocating / symbol not found

[Andrei Pozolotin]

Rich, Alexander:

On 06/24/2016 05:17 PM, Alexander Monakov wrote:
> On Fri, 24 Jun 2016, Rich Felker wrote:
>> OK, this is likely the root of the problem: invalid code assuming that
>> it can load libraries with undefined symbols as long as it doesn't try
>> to use those code paths.
> No, this does not explain the issue: as the first message in thread shows, the
> symbol is defined in libgobject, and if it wasn't, ldd would detect that. So
> we still don't have the root cause.
>
> Alexander
please let me know what else beyond [ldd, scanelf, strace]
can I do to find out what in fact is going on?

Re: musl ldd: swt build: Error relocating / symbol not found

[Rich Felker]

On Fri, Jun 24, 2016 at 08:17:00PM +0300, Alexander Monakov wrote:
> On Fri, 24 Jun 2016, Rich Felker wrote:
> > OK, this is likely the root of the problem: invalid code assuming that
> > it can load libraries with undefined symbols as long as it doesn't try
> > to use those code paths.
> 
> No, this does not explain the issue: as the first message in thread shows, the
> symbol is defined in libgobject, and if it wasn't, ldd would detect that. So
> we still don't have the root cause.

Oh, thanks for catching that... Sorry for the noise.

Rich