some odd library loading errors

some odd library loading errors

[u-uy74]

Hello,

The environment:
 Linux 3.# on x#86
 binutils 2.24
 musl 1.1.8 compiled for i486
 gcc 5.2.0

(the gcc has arch=i486 as the default)

No oddities were observed with the above until trying to use libatomic
which is provided by gcc-5.2.0.

The test case:

$ cat >a.c <<____
int main(){ return 0; }
____

$ gcc -o a a.c -L<path-to-gcc-5.2.0-libs> -latomic

$ LD_LIBRARY_PATH=<path-to-gcc-5.2.0-libs> ./a
Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_store_8: symbol not found
Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_exchange_8: symbol not found
Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_load_8: symbol not found
Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_compare_exchange_8: symbol not found

$

(No error if doing the same with e.g. -lgcc_s.)

Of course the symbols shown in the diagnostics *are* defined in the library.

When I look at the store_8_.o object file, I notice though that the
__atomic_store_8 is not "a function" even though the value is
the same as of select_store_8:

$ objdump -t store_8_.o

store_8_.o:     file format elf32-i386

SYMBOL TABLE:
00000000 l    df *ABS*  00000000 store_n.c
00000000 l    d  .text  00000000 .text
00000000 l    d  .data  00000000 .data
00000000 l    d  .bss   00000000 .bss
00000000 l    d  .text.unlikely 00000000 .text.unlikely
00000030 l     F .text  00000016 select_store_8
00000000 l    d  .debug_info    00000000 .debug_info
00000000 l    d  .debug_abbrev  00000000 .debug_abbrev
00000000 l    d  .debug_loc     00000000 .debug_loc
00000000 l    d  .debug_aranges 00000000 .debug_aranges
00000000 l    d  .debug_ranges  00000000 .debug_ranges
00000000 l    d  .debug_line    00000000 .debug_line
00000000 l    d  .debug_str     00000000 .debug_str
00000000 l    d  .note.GNU-stack        00000000 .note.GNU-stack
00000000 l    d  .eh_frame      00000000 .eh_frame
00000000 l    d  .comment       00000000 .comment
00000000 g     F .text  0000002c .hidden libat_store_8
00000000         *UND*  00000000 .hidden libat_lock_1
00000000         *UND*  00000000 .hidden libat_unlock_1
00000000         *UND*  00000000 .hidden libat_feat1_edx
00000000         *UND*  00000000 .hidden libat_store_8_i1
00000030 g       .text  00000016 __atomic_store_8


$

I would appreciate help with finding out who is doing wrong:
- gcc
- ld
- musl
- myself
and what is to be corrected to make it work.

The actual practical problem is with the mesa library not being able
to load drivers which need symbols from libatomic, but this tiny test
case ought to be working, so it is not mesa who is at fault.
This does not either seem to involve lazy binding, which is otherwise
a known source of problems for mesa.

It looks like I can avoid the usage of libatomic with
 "gcc -fno-sync-libcalls"
but the behaviour seems wrong and I would rather see it fixed or otherwise
learn if I made an error.

Regards,
Rune

Re: some odd library loading errors

[Szabolcs Nagy]

* u-uy74@aetey.se <u-uy74@aetey.se> [2015-11-17 13:14:29 +0100]:
> The environment:
>  Linux 3.# on x#86
>  binutils 2.24
>  musl 1.1.8 compiled for i486
>  gcc 5.2.0
> 
> (the gcc has arch=i486 as the default)
> 
> No oddities were observed with the above until trying to use libatomic
> which is provided by gcc-5.2.0.
> 
> The test case:
> 
> $ cat >a.c <<____
> int main(){ return 0; }
> ____
> 
> $ gcc -o a a.c -L<path-to-gcc-5.2.0-libs> -latomic
> 
> $ LD_LIBRARY_PATH=<path-to-gcc-5.2.0-libs> ./a
> Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_store_8: symbol not found
> Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_exchange_8: symbol not found
> Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_load_8: symbol not found
> Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_compare_exchange_8: symbol not found
> 

this is a known issue, they use the gnu ifunc extension of
elf to dispatch between different implementations based on
the machine at library loadtime.

> I would appreciate help with finding out who is doing wrong:
> - gcc
> - ld
> - musl
> - myself
> and what is to be corrected to make it work.
> 

build gcc with --disable-gnu-indirect-function

(then i guess libatomic will always use the
portable implementation.)

Re: some odd library loading errors

[Rich Felker]

On Tue, Nov 17, 2015 at 04:23:56PM +0100, Szabolcs Nagy wrote:
> * u-uy74@aetey.se <u-uy74@aetey.se> [2015-11-17 13:14:29 +0100]:
> > The environment:
> >  Linux 3.# on x#86
> >  binutils 2.24
> >  musl 1.1.8 compiled for i486
> >  gcc 5.2.0
> > 
> > (the gcc has arch=i486 as the default)
> > 
> > No oddities were observed with the above until trying to use libatomic
> > which is provided by gcc-5.2.0.
> > 
> > The test case:
> > 
> > $ cat >a.c <<____
> > int main(){ return 0; }
> > ____
> > 
> > $ gcc -o a a.c -L<path-to-gcc-5.2.0-libs> -latomic
> > 
> > $ LD_LIBRARY_PATH=<path-to-gcc-5.2.0-libs> ./a
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_store_8: symbol not found
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_exchange_8: symbol not found
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_load_8: symbol not found
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_compare_exchange_8: symbol not found
> > 
> 
> this is a known issue, they use the gnu ifunc extension of
> elf to dispatch between different implementations based on
> the machine at library loadtime.

Is there something we need to add to the musl patches to disable this?
I would like to make sure both that it's going upstream and that I
have a patch in my toolchain build repo.

Rich

Re: some odd library loading errors

[u-uy74]

On Tue, Nov 17, 2015 at 04:23:56PM +0100, Szabolcs Nagy wrote:
> > $ LD_LIBRARY_PATH=<path-to-gcc-5.2.0-libs> ./a
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_store_8: symbol not found
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_exchange_8: symbol not found
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_load_8: symbol not found
> > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_compare_exchange_8: symbol not found
> > 
> 
> this is a known issue, they use the gnu ifunc extension of
> elf to dispatch between different implementations based on
> the machine at library loadtime.

Oh. Now I see. Indeed you mentioned this earlier.
Bad style gcc.

> build gcc with --disable-gnu-indirect-function

> (then i guess libatomic will always use the
> portable implementation.)

Sigh. Thanks Szabolcs!

Rune

Re: some odd library loading errors

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2015-11-17 10:27:28 -0500]:
> On Tue, Nov 17, 2015 at 04:23:56PM +0100, Szabolcs Nagy wrote:
> > * u-uy74@aetey.se <u-uy74@aetey.se> [2015-11-17 13:14:29 +0100]:
> > > The environment:
> > >  Linux 3.# on x#86
> > >  binutils 2.24
> > >  musl 1.1.8 compiled for i486
> > >  gcc 5.2.0
> > > 
> > > (the gcc has arch=i486 as the default)
> > > 
> > > No oddities were observed with the above until trying to use libatomic
> > > which is provided by gcc-5.2.0.
> > > 
> > > The test case:
> > > 
> > > $ cat >a.c <<____
> > > int main(){ return 0; }
> > > ____
> > > 
> > > $ gcc -o a a.c -L<path-to-gcc-5.2.0-libs> -latomic
> > > 
> > > $ LD_LIBRARY_PATH=<path-to-gcc-5.2.0-libs> ./a
> > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_store_8: symbol not found
> > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_exchange_8: symbol not found
> > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_load_8: symbol not found
> > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_compare_exchange_8: symbol not found
> > > 
> > 
> > this is a known issue, they use the gnu ifunc extension of
> > elf to dispatch between different implementations based on
> > the machine at library loadtime.
> 
> Is there something we need to add to the musl patches to disable this?
> I would like to make sure both that it's going upstream and that I
> have a patch in my toolchain build repo.
> 

i guess --disable-gnu-indirect-function could be the default when
gcc targets musl (assuming musl will not implement ifuncs anytime soon).

Re: some odd library loading errors

[Rich Felker]

On Tue, Nov 17, 2015 at 04:45:16PM +0100, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2015-11-17 10:27:28 -0500]:
> > On Tue, Nov 17, 2015 at 04:23:56PM +0100, Szabolcs Nagy wrote:
> > > * u-uy74@aetey.se <u-uy74@aetey.se> [2015-11-17 13:14:29 +0100]:
> > > > The environment:
> > > >  Linux 3.# on x#86
> > > >  binutils 2.24
> > > >  musl 1.1.8 compiled for i486
> > > >  gcc 5.2.0
> > > > 
> > > > (the gcc has arch=i486 as the default)
> > > > 
> > > > No oddities were observed with the above until trying to use libatomic
> > > > which is provided by gcc-5.2.0.
> > > > 
> > > > The test case:
> > > > 
> > > > $ cat >a.c <<____
> > > > int main(){ return 0; }
> > > > ____
> > > > 
> > > > $ gcc -o a a.c -L<path-to-gcc-5.2.0-libs> -latomic
> > > > 
> > > > $ LD_LIBRARY_PATH=<path-to-gcc-5.2.0-libs> ./a
> > > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_store_8: symbol not found
> > > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_exchange_8: symbol not found
> > > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_load_8: symbol not found
> > > > Error relocating <path-to-gcc-5.2.0-libs>/libatomic.so.1: __atomic_compare_exchange_8: symbol not found
> > > > 
> > > 
> > > this is a known issue, they use the gnu ifunc extension of
> > > elf to dispatch between different implementations based on
> > > the machine at library loadtime.
> > 
> > Is there something we need to add to the musl patches to disable this?
> > I would like to make sure both that it's going upstream and that I
> > have a patch in my toolchain build repo.
> > 
> 
> i guess --disable-gnu-indirect-function could be the default when
> gcc targets musl (assuming musl will not implement ifuncs anytime soon).

I think that's a reasonable assumption. Based on what I've seen
lately, rather than finding solutions to the problems we knew about
already, the rabbit hole keeps going deeper...

If you have reason to believe otherwise could you explain how we might
reasonably support ifunc?

Rich

Re: some odd library loading errors

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2015-11-17 10:55:45 -0500]:
> On Tue, Nov 17, 2015 at 04:45:16PM +0100, Szabolcs Nagy wrote:
> > 
> > i guess --disable-gnu-indirect-function could be the default when
> > gcc targets musl (assuming musl will not implement ifuncs anytime soon).
> 
> I think that's a reasonable assumption. Based on what I've seen
> lately, rather than finding solutions to the problems we knew about
> already, the rabbit hole keeps going deeper...
> 
> If you have reason to believe otherwise could you explain how we might
> reasonably support ifunc?
> 

well we can add support for ifunc by..

calling the ifunc resolver during reloc processing when STT_GNU_IFUNC
symbol or R_*_IRELATIVE reloc is found and in case of static linking
the relocs between __rel_iplt_{start,end} should be processed somehow.

the only ugliness is that passing arguments to the resolver is arch
specific.. and that there is no guarantee what the resolver might do
(but that's the same with glibc and it is something the user should
worry about)

i think even if musl does the reloc processing in different order
than glibc, this might work in practice and if the resolver crashes
we can say that it invoked ub.

Re: some odd library loading errors

[Rich Felker]

On Tue, Nov 17, 2015 at 05:34:00PM +0100, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2015-11-17 10:55:45 -0500]:
> > On Tue, Nov 17, 2015 at 04:45:16PM +0100, Szabolcs Nagy wrote:
> > > 
> > > i guess --disable-gnu-indirect-function could be the default when
> > > gcc targets musl (assuming musl will not implement ifuncs anytime soon).
> > 
> > I think that's a reasonable assumption. Based on what I've seen
> > lately, rather than finding solutions to the problems we knew about
> > already, the rabbit hole keeps going deeper...
> > 
> > If you have reason to believe otherwise could you explain how we might
> > reasonably support ifunc?
> > 
> 
> well we can add support for ifunc by..
> 
> calling the ifunc resolver during reloc processing when STT_GNU_IFUNC
> symbol or R_*_IRELATIVE reloc is found and in case of static linking
> the relocs between __rel_iplt_{start,end} should be processed somehow.

This is probably a significant code size burden for static linking
unless there's some way to avoid linking it when ifunc is not used.
There's also the static-pie case you haven't considered, which would
require its own separate implementation. Presumably that would be
processing R_*_IRELATIVE since symbolic relocations are not permitted
here.

> the only ugliness is that passing arguments to the resolver is arch
> specific.. and that there is no guarantee what the resolver might do
> (but that's the same with glibc and it is something the user should
> worry about)

Well that's ugly too.

> i think even if musl does the reloc processing in different order
> than glibc, this might work in practice and if the resolver crashes
> we can say that it invoked ub.

It seems to me that order issues, except for circular dependencies,
are less of an issue when you have a lazy resolver; if A gets resolved
first but A needs B, then calling B invokes the lazy resolver and all
is fine. Since musl doesn't (and won't) have lazy resolver, the whole
setup seems much more fragile.

Then of course there's just the general issue of having arbitrary code
run in callbacks called from the dynamic linker with things in
inconsistent state. Without a formal spec for what ifunc resolvers are
actually allowed to do, we can easily fall into a trap of "having to"
support whatever happened to work on one particular version of the
implementation.

Rich