musl can't handle gold's STB_LOCAL TLS symbols

musl can't handle gold's STB_LOCAL TLS symbols

[Ryan Prichard]

[-- Attachment #1: Type: text/plain, Size: 1767 bytes --]

I noticed that if I use a version script with ld.gold to make a TLS symbol
in a DSO local, the symbol remains in the .dynsym table but with local
binding. Gold emits DTPMOD/DTPOFF relocations to the local TLS symbol, and
when musl tries to load the DSO, it apparently tries to look up the symbol
name globally and fails.

$ cat vers
{
global:
  get_tls_var;
local:
  *;
};

$ cat dso.c
__thread int tlsvar;
int get_tls_var(void) {
  return tlsvar;
}

$ cat main.c
int get_tls_var(void);
int main() {
  get_tls_var();
  return 0;
}

$ musl-gcc dso.c -fpic -shared -fuse-ld=gold -Wl,-version-script=vers -o
libdso.so
$ musl-gcc main.c libdso.so -Wl,-rpath,'$ORIGIN' -o main
$ ./main
Error relocating /tmp/local-tls-symbol/libdso.so: tlsvar: symbol not found
Error relocating /tmp/local-tls-symbol/libdso.so: tlsvar: symbol not found

$ readelf -rW --dyn-syms libdso.so
...
0000000000001fd8  0000000100000010 R_X86_64_DTPMOD64      0000000000000000
tlsvar + 0
0000000000001fe0  0000000100000011 R_X86_64_DTPOFF64      0000000000000000
tlsvar + 0
...
     1: 0000000000000000     4 TLS     LOCAL  DEFAULT   13 tlsvar
...

The test program works with ld.bfd, because ld.bfd converts the DTPMOD
relocation to 0 and omits the DTPOFF relocation. There was a somewhat
similar issue with gold+musl involving a DTPMOD relocation to a
local section symbol, https://sourceware.org/bugzilla/show_bug.cgi?id=17699.
That issue prompted a thread on the generic-abi group,
https://groups.google.com/d/topic/generic-abi/dJ4_Y78aQ2M/discussion.

I'm wondering if this problem is a bug in musl or gold. I also wonder if
DTPOFF can reference a TLS section, even though the value of a TLS section
symbol isn't suitable for DTPOFF unless it's first adjusted by the
segment's p_vaddr.

-Ryan

[-- Attachment #2: Type: text/html, Size: 2338 bytes --]

Re: musl can't handle gold's STB_LOCAL TLS symbols

[Rich Felker]

On Thu, May 23, 2019 at 05:47:13PM -0700, Ryan Prichard wrote:
> I noticed that if I use a version script with ld.gold to make a TLS symbol
> in a DSO local, the symbol remains in the .dynsym table but with local
> binding. Gold emits DTPMOD/DTPOFF relocations to the local TLS symbol, and
> when musl tries to load the DSO, it apparently tries to look up the symbol
> name globally and fails.
> 
> $ cat vers
> {
> global:
>   get_tls_var;
> local:
>   *;
> };
> 
> $ cat dso.c
> __thread int tlsvar;
> int get_tls_var(void) {
>   return tlsvar;
> }
> 
> $ cat main.c
> int get_tls_var(void);
> int main() {
>   get_tls_var();
>   return 0;
> }
> 
> $ musl-gcc dso.c -fpic -shared -fuse-ld=gold -Wl,-version-script=vers -o
> libdso.so
> $ musl-gcc main.c libdso.so -Wl,-rpath,'$ORIGIN' -o main
> $ ./main
> Error relocating /tmp/local-tls-symbol/libdso.so: tlsvar: symbol not found
> Error relocating /tmp/local-tls-symbol/libdso.so: tlsvar: symbol not found
> 
> $ readelf -rW --dyn-syms libdso.so
> ....
> 0000000000001fd8  0000000100000010 R_X86_64_DTPMOD64      0000000000000000
> tlsvar + 0
> 0000000000001fe0  0000000100000011 R_X86_64_DTPOFF64      0000000000000000
> tlsvar + 0
> ....
>      1: 0000000000000000     4 TLS     LOCAL  DEFAULT   13 tlsvar
> ....
> 
> The test program works with ld.bfd, because ld.bfd converts the DTPMOD
> relocation to 0 and omits the DTPOFF relocation. There was a somewhat
> similar issue with gold+musl involving a DTPMOD relocation to a
> local section symbol, https://sourceware.org/bugzilla/show_bug.cgi?id=17699.
> That issue prompted a thread on the generic-abi group,
> https://groups.google.com/d/topic/generic-abi/dJ4_Y78aQ2M/discussion.
> 
> I'm wondering if this problem is a bug in musl or gold. I also wonder if

I would consider this a bug in gold. There is no reason to leave local
symbols unresolved until runtime; resolving them is ld's whole job.

> DTPOFF can reference a TLS section, even though the value of a TLS section
> symbol isn't suitable for DTPOFF unless it's first adjusted by the
> segment's p_vaddr.

I don't see a good reason for it to reference a section either. It
should just have a 0 symbol reference, and store the ld-determined
offset to the object in the addend. Any kind of symbolic reference
here is just going to be a waste of time doing a lookup at runtime.

Rich

Re: musl can't handle gold's STB_LOCAL TLS symbols

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2019-05-23 21:12:04 -0400]:
> On Thu, May 23, 2019 at 05:47:13PM -0700, Ryan Prichard wrote:
> > The test program works with ld.bfd, because ld.bfd converts the DTPMOD
> > relocation to 0 and omits the DTPOFF relocation. There was a somewhat
> > similar issue with gold+musl involving a DTPMOD relocation to a
> > local section symbol, https://sourceware.org/bugzilla/show_bug.cgi?id=17699.
> > That issue prompted a thread on the generic-abi group,
> > https://groups.google.com/d/topic/generic-abi/dJ4_Y78aQ2M/discussion.
> > 
> > I'm wondering if this problem is a bug in musl or gold. I also wonder if
> 
> I would consider this a bug in gold. There is no reason to leave local
> symbols unresolved until runtime; resolving them is ld's whole job.
> 
> > DTPOFF can reference a TLS section, even though the value of a TLS section
> > symbol isn't suitable for DTPOFF unless it's first adjusted by the
> > segment's p_vaddr.
> 
> I don't see a good reason for it to reference a section either. It
> should just have a 0 symbol reference, and store the ld-determined
> offset to the object in the addend. Any kind of symbolic reference
> here is just going to be a waste of time doing a lookup at runtime.

the sysv elf spec allows leaving STB_LOCAL symbols
in the dynamic symbol table if they were hidden [1].
it does not say if symbolic dynamic relocs may refer
to them, though.

i think it should not be too hard to adjust
do_relocs to handle this, not sure about do_dlsym.
current dlsym does not seem to check OK_BIND
so a local sym may preempt a global one.

i think allowing plt relocs for local syms can have
unexpected effects, so i don't think that would work
reliably (on systems that support lazy binding, ifuncs
ldaudit and various other plt related hacks).

[1] http://www.sco.com/developers/gabi/latest/ch4.symtab.html#visibility

Re: musl can't handle gold's STB_LOCAL TLS symbols

[Rich Felker]

On Sat, May 25, 2019 at 11:18:16AM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2019-05-23 21:12:04 -0400]:
> > On Thu, May 23, 2019 at 05:47:13PM -0700, Ryan Prichard wrote:
> > > The test program works with ld.bfd, because ld.bfd converts the DTPMOD
> > > relocation to 0 and omits the DTPOFF relocation. There was a somewhat
> > > similar issue with gold+musl involving a DTPMOD relocation to a
> > > local section symbol, https://sourceware.org/bugzilla/show_bug.cgi?id=17699.
> > > That issue prompted a thread on the generic-abi group,
> > > https://groups.google.com/d/topic/generic-abi/dJ4_Y78aQ2M/discussion.
> > > 
> > > I'm wondering if this problem is a bug in musl or gold. I also wonder if
> > 
> > I would consider this a bug in gold. There is no reason to leave local
> > symbols unresolved until runtime; resolving them is ld's whole job.
> > 
> > > DTPOFF can reference a TLS section, even though the value of a TLS section
> > > symbol isn't suitable for DTPOFF unless it's first adjusted by the
> > > segment's p_vaddr.
> > 
> > I don't see a good reason for it to reference a section either. It
> > should just have a 0 symbol reference, and store the ld-determined
> > offset to the object in the addend. Any kind of symbolic reference
> > here is just going to be a waste of time doing a lookup at runtime.
> 
> the sysv elf spec allows leaving STB_LOCAL symbols
> in the dynamic symbol table if they were hidden [1].
> it does not say if symbolic dynamic relocs may refer
> to them, though.

I don't think there's any reasonable way a reference to the symbol
from a relocation can distinguish between one that needs to use the
local definition and one that needs to follow normal global symbol
resolution, is there? Or can it just skip the lookup entirely and use
the reference as the definition in this case?? If that's possible it
might be trivial and non-controversial to support.

> i think it should not be too hard to adjust
> do_relocs to handle this, not sure about do_dlsym.
> current dlsym does not seem to check OK_BIND
> so a local sym may preempt a global one.

This is a bug we need to fix. It was probably fixed by your old patch
to unify symbol lookup paths, but that bitrotted before it ever got
applied.

Rich

Re: musl can't handle gold's STB_LOCAL TLS symbols

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2019-05-25 12:51:33 -0400]:
> On Sat, May 25, 2019 at 11:18:16AM +0200, Szabolcs Nagy wrote:
> > 
> > the sysv elf spec allows leaving STB_LOCAL symbols
> > in the dynamic symbol table if they were hidden [1].
> > it does not say if symbolic dynamic relocs may refer
> > to them, though.
> 
> I don't think there's any reasonable way a reference to the symbol
> from a relocation can distinguish between one that needs to use the
> local definition and one that needs to follow normal global symbol
> resolution, is there? Or can it just skip the lookup entirely and use
> the reference as the definition in this case?? If that's possible it
> might be trivial and non-controversial to support.

i think using the ref as the def is the only sensible
interpretation for STB_LOCAL.

glibc seems to do that but there are some warts: on ppc32
sym->st_value is already added to reloc->r_addend for
STB_LOCAL syms, but not for tls syms (assuming glibc is right).