armv7-m segv when throwing c++ exception

armv7-m segv when throwing c++ exception

[Patrick Oppenlander]

Hi all,

I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
using musl-cross-make for armv7-m.

% cat test.cpp
int main()
{
        throw 0;
}
% armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
% qemu-arm ./a.out
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
[2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out

We should be seeing something like "terminate called after throwing an
instance of 'int'".

I have an ODROID-C2 here which is a convenient place to run gdb:

$ gdb -q ./a.out
Reading symbols from ./a.out...
(gdb) run
Program received signal SIGSEGV, Segmentation fault.
0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
    at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
file or directory.
(gdb) bt
#0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
    at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
#1  0xf77cf902 in main () at test.cpp:3
(gdb) disas
Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
std::type_info*, void (*)(void*)):
   0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
   0xf77cf9e2 <+2>: mov r5, r0
   0xf77cf9e4 <+4>: mov r7, r1
   0xf77cf9e6 <+6>: mov r6, r2
   0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
   0xf77cf9ec <+12>: mov r4, r0
   0xf77cf9ee <+14>: mov r1, r7
   0xf77cf9f0 <+16>: mov r2, r6
   0xf77cf9f2 <+18>: mov r0, r5
=> 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
   0xf77cf9f6 <+22>: adds r3, #1
   0xf77cf9f8 <+24>: str r3, [r4, #4]
   0xf77cf9fa <+26>: bl 0xf77cf984
<__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
(*)(void*))>
   0xf77cf9fe <+30>: movs r3, #1
   0xf77cfa00 <+32>: mov r4, r0
   0xf77cfa02 <+34>: str.w r3, [r4], #40
   0xf77cfa06 <+38>: mov r0, r4
   0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
   0xf77cfa0c <+44>: mov r0, r4
   0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
   0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
End of assembler dump.
(gdb) p/x $r4
$1 = 0x1

That doesn't look good.

(gdb) disas __cxa_get_globals
Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
   0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
<__cxxabiv1::__cxa_get_globals()+16>)
   0xf77d01c6 <+2>: push {r3, lr}
   0xf77d01c8 <+4>: add r0, pc
   0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
   0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
<__cxxabiv1::__cxa_get_globals()+20>)
   0xf77d01d0 <+12>: add r0, r3
   0xf77d01d2 <+14>: pop {r3, pc}
   0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
   0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
End of assembler dump.

OK, so what did __tls_get_addr return?

(gdb) b *(__cxa_get_globals + 10)
Breakpoint 3 at 0xf77d01ce: file
../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
(gdb) run
Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
file or directory.
(gdb) p/x $r0
$3 = 0x1

That's doesn't look good either. Let's look at __tls_get_addr.

void *__tls_get_addr(tls_mod_off_t *v)
{
        pthread_t self = __pthread_self();
        return (void *)(self->dtv[v[0]] + v[1]);
}

(gdb) disas __tls_get_addr (annotations added by me)
Dump of assembler code for function __tls_get_addr:
r2 = v[0]
   0x0000c940 <+0>: ldr r2, [r0, #0]
r3 = tls
   0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
self = r3 - 120
r3 = *(r3 - 116) = self->dtv
   0x0000c946 <+6>: ldr.w r3, [r3, #-116]
r0 = v[1]
   0x0000c94a <+10>: ldr r0, [r0, #4]
r2 = *(r3 + r2 * 4) = dtv[r2]
   0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
r0 = r0 + r2
   0x0000c950 <+16>: add r0, r2
   0x0000c952 <+18>: bx lr
End of assembler dump.
(gdb) b __tls_get_addr

So after pulling out the values I end up with:
v = 0xf77f000c
0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
v[0] = 0
v[1] = 0
self = <builtin_tls> 0xf77f064c
0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
self->dtv = <builtin_tls+144> 0xf77f06dc
0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
return = dtv[v[0]] + v[1] = 1 + 0 = 1

At this point I'm out of my depth.

Where should I look next?

Thanks,

Patrick

Re: armv7-m segv when throwing c++ exception

[Szabolcs Nagy]

* Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> Hi all,
> 
> I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> using musl-cross-make for armv7-m.
> 
> % cat test.cpp
> int main()
> {
>         throw 0;
> }
> % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> % qemu-arm ./a.out
> qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> 
> We should be seeing something like "terminate called after throwing an
> instance of 'int'".

there is at least one known issue in this area:
libgcc uses pthread apis via weak refs so with static
linking they don't get linked in, however i think that
should only affect unwinding in multi-thread processes,
so this is something else

> I have an ODROID-C2 here which is a convenient place to run gdb:
> 
> $ gdb -q ./a.out
> Reading symbols from ./a.out...
> (gdb) run
> Program received signal SIGSEGV, Segmentation fault.
> 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
>     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> file or directory.
> (gdb) bt
> #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
>     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> #1  0xf77cf902 in main () at test.cpp:3
> (gdb) disas
> Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> std::type_info*, void (*)(void*)):
>    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
>    0xf77cf9e2 <+2>: mov r5, r0
>    0xf77cf9e4 <+4>: mov r7, r1
>    0xf77cf9e6 <+6>: mov r6, r2
>    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
>    0xf77cf9ec <+12>: mov r4, r0
>    0xf77cf9ee <+14>: mov r1, r7
>    0xf77cf9f0 <+16>: mov r2, r6
>    0xf77cf9f2 <+18>: mov r0, r5
> => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
>    0xf77cf9f6 <+22>: adds r3, #1
>    0xf77cf9f8 <+24>: str r3, [r4, #4]
>    0xf77cf9fa <+26>: bl 0xf77cf984
> <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> (*)(void*))>
>    0xf77cf9fe <+30>: movs r3, #1
>    0xf77cfa00 <+32>: mov r4, r0
>    0xf77cfa02 <+34>: str.w r3, [r4], #40
>    0xf77cfa06 <+38>: mov r0, r4
>    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
>    0xf77cfa0c <+44>: mov r0, r4
>    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
>    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> End of assembler dump.
> (gdb) p/x $r4
> $1 = 0x1
> 
> That doesn't look good.
> 
> (gdb) disas __cxa_get_globals
> Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
>    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> <__cxxabiv1::__cxa_get_globals()+16>)
>    0xf77d01c6 <+2>: push {r3, lr}
>    0xf77d01c8 <+4>: add r0, pc
>    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
>    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> <__cxxabiv1::__cxa_get_globals()+20>)
>    0xf77d01d0 <+12>: add r0, r3
>    0xf77d01d2 <+14>: pop {r3, pc}
>    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
>    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> End of assembler dump.

interesting, i thought the linker would relax __tls_get_addr
calls in static linked executables to local-exec access model,
but maybe that's not implemented on arm.

can you look at the readelf -aW a.out and see if the linker
left any dynamic tls relocs in the exe?

> 
> OK, so what did __tls_get_addr return?
> 
> (gdb) b *(__cxa_get_globals + 10)
> Breakpoint 3 at 0xf77d01ce: file
> ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> (gdb) run
> Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> file or directory.
> (gdb) p/x $r0
> $3 = 0x1
> 
> That's doesn't look good either. Let's look at __tls_get_addr.
> 
> void *__tls_get_addr(tls_mod_off_t *v)
> {
>         pthread_t self = __pthread_self();
>         return (void *)(self->dtv[v[0]] + v[1]);
> }
> 
> (gdb) disas __tls_get_addr (annotations added by me)
> Dump of assembler code for function __tls_get_addr:
> r2 = v[0]
>    0x0000c940 <+0>: ldr r2, [r0, #0]
> r3 = tls
>    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> self = r3 - 120
> r3 = *(r3 - 116) = self->dtv
>    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> r0 = v[1]
>    0x0000c94a <+10>: ldr r0, [r0, #4]
> r2 = *(r3 + r2 * 4) = dtv[r2]
>    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> r0 = r0 + r2
>    0x0000c950 <+16>: add r0, r2
>    0x0000c952 <+18>: bx lr
> End of assembler dump.
> (gdb) b __tls_get_addr
> 
> So after pulling out the values I end up with:
> v = 0xf77f000c
> 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> v[0] = 0
> v[1] = 0
> self = <builtin_tls> 0xf77f064c
> 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> self->dtv = <builtin_tls+144> 0xf77f06dc
> 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> return = dtv[v[0]] + v[1] = 1 + 0 = 1
> 
> At this point I'm out of my depth.

dtv[0] is special and just stores the length of the dtv
(which is correctly 1, meaning there is 1 elf module with tls).

so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
the tls address of the executable) i think v[0] should be set
up by the static linker so it may be a binutils bug.
(or maybe static linking is special and then dtv[0] should be
set equal to dtv[1]?)

> 
> Where should I look next?
> 
> Thanks,
> 
> Patrick

Re: armv7-m segv when throwing c++ exception

[Rich Felker]

On Wed, Oct 02, 2019 at 11:22:33AM +0200, Szabolcs Nagy wrote:
> * Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> > Hi all,
> > 
> > I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> > using musl-cross-make for armv7-m.
> > 
> > % cat test.cpp
> > int main()
> > {
> >         throw 0;
> > }
> > % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> > % qemu-arm ./a.out
> > qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> > 
> > We should be seeing something like "terminate called after throwing an
> > instance of 'int'".
> 
> there is at least one known issue in this area:
> libgcc uses pthread apis via weak refs so with static
> linking they don't get linked in, however i think that
> should only affect unwinding in multi-thread processes,
> so this is something else
> 
> > I have an ODROID-C2 here which is a convenient place to run gdb:
> > 
> > $ gdb -q ./a.out
> > Reading symbols from ./a.out...
> > (gdb) run
> > Program received signal SIGSEGV, Segmentation fault.
> > 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> > file or directory.
> > (gdb) bt
> > #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > #1  0xf77cf902 in main () at test.cpp:3
> > (gdb) disas
> > Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> > std::type_info*, void (*)(void*)):
> >    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
> >    0xf77cf9e2 <+2>: mov r5, r0
> >    0xf77cf9e4 <+4>: mov r7, r1
> >    0xf77cf9e6 <+6>: mov r6, r2
> >    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
> >    0xf77cf9ec <+12>: mov r4, r0
> >    0xf77cf9ee <+14>: mov r1, r7
> >    0xf77cf9f0 <+16>: mov r2, r6
> >    0xf77cf9f2 <+18>: mov r0, r5
> > => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
> >    0xf77cf9f6 <+22>: adds r3, #1
> >    0xf77cf9f8 <+24>: str r3, [r4, #4]
> >    0xf77cf9fa <+26>: bl 0xf77cf984
> > <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> > (*)(void*))>
> >    0xf77cf9fe <+30>: movs r3, #1
> >    0xf77cfa00 <+32>: mov r4, r0
> >    0xf77cfa02 <+34>: str.w r3, [r4], #40
> >    0xf77cfa06 <+38>: mov r0, r4
> >    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
> >    0xf77cfa0c <+44>: mov r0, r4
> >    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
> >    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> > End of assembler dump.
> > (gdb) p/x $r4
> > $1 = 0x1
> > 
> > That doesn't look good.
> > 
> > (gdb) disas __cxa_get_globals
> > Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
> >    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> > <__cxxabiv1::__cxa_get_globals()+16>)
> >    0xf77d01c6 <+2>: push {r3, lr}
> >    0xf77d01c8 <+4>: add r0, pc
> >    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
> >    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> > <__cxxabiv1::__cxa_get_globals()+20>)
> >    0xf77d01d0 <+12>: add r0, r3
> >    0xf77d01d2 <+14>: pop {r3, pc}
> >    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
> >    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> > End of assembler dump.
> 
> interesting, i thought the linker would relax __tls_get_addr
> calls in static linked executables to local-exec access model,
> but maybe that's not implemented on arm.
> 
> can you look at the readelf -aW a.out and see if the linker
> left any dynamic tls relocs in the exe?
> 
> > 
> > OK, so what did __tls_get_addr return?
> > 
> > (gdb) b *(__cxa_get_globals + 10)
> > Breakpoint 3 at 0xf77d01ce: file
> > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> > (gdb) run
> > Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> > 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> > file or directory.
> > (gdb) p/x $r0
> > $3 = 0x1
> > 
> > That's doesn't look good either. Let's look at __tls_get_addr.
> > 
> > void *__tls_get_addr(tls_mod_off_t *v)
> > {
> >         pthread_t self = __pthread_self();
> >         return (void *)(self->dtv[v[0]] + v[1]);
> > }
> > 
> > (gdb) disas __tls_get_addr (annotations added by me)
> > Dump of assembler code for function __tls_get_addr:
> > r2 = v[0]
> >    0x0000c940 <+0>: ldr r2, [r0, #0]
> > r3 = tls
> >    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> > self = r3 - 120
> > r3 = *(r3 - 116) = self->dtv
> >    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> > r0 = v[1]
> >    0x0000c94a <+10>: ldr r0, [r0, #4]
> > r2 = *(r3 + r2 * 4) = dtv[r2]
> >    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> > r0 = r0 + r2
> >    0x0000c950 <+16>: add r0, r2
> >    0x0000c952 <+18>: bx lr
> > End of assembler dump.
> > (gdb) b __tls_get_addr
> > 
> > So after pulling out the values I end up with:
> > v = 0xf77f000c
> > 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> > v[0] = 0
> > v[1] = 0
> > self = <builtin_tls> 0xf77f064c
> > 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> > self->dtv = <builtin_tls+144> 0xf77f06dc
> > 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> > return = dtv[v[0]] + v[1] = 1 + 0 = 1
> > 
> > At this point I'm out of my depth.
> 
> dtv[0] is special and just stores the length of the dtv
> (which is correctly 1, meaning there is 1 elf module with tls).
> 
> so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
> the tls address of the executable) i think v[0] should be set
> up by the static linker so it may be a binutils bug.
> (or maybe static linking is special and then dtv[0] should be
> set equal to dtv[1]?)

MIPS had the exact same bug; I think it's fixed upstream now since we
had the patch for binutils 2.27 in mcm but dropped it in 2.32:

https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.27/0004-mips-pie-tls.diff

It affects static PIE only, so -no-pie should be able to work around
it (conditional for setting the 1 was essentially !ET_DYN instead of
is_executable).

We could work around this in musl in one of two ways: rcrt1 could
process DTPMOD relocs and write a 1, or the static version of
__init_tls could just duplicate dtv[1] in dtv[0] (the latter is only
valid up until we add static-linked dlopen, if we ever do that).
However I was fairly strongly against doing that for MIPS, since it
was a binutils bug and only affected static linking (so it's not a
runtime ABI issue; the binary either works at link-time or it doesn't)
and I'd lean towards treating ARM the same.

It should be easy to make and apply the binutils patch to fix this,
and shouldn't be too hard to grep for the same bug in all archs (since
binutils nicely duplicates this logic for every single arch, for no
reason whatsoever).

Rich

Re: armv7-m segv when throwing c++ exception

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 7645 bytes --]

On Wed, Oct 02, 2019 at 07:40:45AM -0400, Rich Felker wrote:
> On Wed, Oct 02, 2019 at 11:22:33AM +0200, Szabolcs Nagy wrote:
> > * Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> > > Hi all,
> > > 
> > > I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> > > using musl-cross-make for armv7-m.
> > > 
> > > % cat test.cpp
> > > int main()
> > > {
> > >         throw 0;
> > > }
> > > % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> > > % qemu-arm ./a.out
> > > qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> > > 
> > > We should be seeing something like "terminate called after throwing an
> > > instance of 'int'".
> > 
> > there is at least one known issue in this area:
> > libgcc uses pthread apis via weak refs so with static
> > linking they don't get linked in, however i think that
> > should only affect unwinding in multi-thread processes,
> > so this is something else
> > 
> > > I have an ODROID-C2 here which is a convenient place to run gdb:
> > > 
> > > $ gdb -q ./a.out
> > > Reading symbols from ./a.out...
> > > (gdb) run
> > > Program received signal SIGSEGV, Segmentation fault.
> > > 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> > > file or directory.
> > > (gdb) bt
> > > #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > #1  0xf77cf902 in main () at test.cpp:3
> > > (gdb) disas
> > > Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> > > std::type_info*, void (*)(void*)):
> > >    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
> > >    0xf77cf9e2 <+2>: mov r5, r0
> > >    0xf77cf9e4 <+4>: mov r7, r1
> > >    0xf77cf9e6 <+6>: mov r6, r2
> > >    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
> > >    0xf77cf9ec <+12>: mov r4, r0
> > >    0xf77cf9ee <+14>: mov r1, r7
> > >    0xf77cf9f0 <+16>: mov r2, r6
> > >    0xf77cf9f2 <+18>: mov r0, r5
> > > => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
> > >    0xf77cf9f6 <+22>: adds r3, #1
> > >    0xf77cf9f8 <+24>: str r3, [r4, #4]
> > >    0xf77cf9fa <+26>: bl 0xf77cf984
> > > <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> > > (*)(void*))>
> > >    0xf77cf9fe <+30>: movs r3, #1
> > >    0xf77cfa00 <+32>: mov r4, r0
> > >    0xf77cfa02 <+34>: str.w r3, [r4], #40
> > >    0xf77cfa06 <+38>: mov r0, r4
> > >    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
> > >    0xf77cfa0c <+44>: mov r0, r4
> > >    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
> > >    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> > > End of assembler dump.
> > > (gdb) p/x $r4
> > > $1 = 0x1
> > > 
> > > That doesn't look good.
> > > 
> > > (gdb) disas __cxa_get_globals
> > > Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
> > >    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> > > <__cxxabiv1::__cxa_get_globals()+16>)
> > >    0xf77d01c6 <+2>: push {r3, lr}
> > >    0xf77d01c8 <+4>: add r0, pc
> > >    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
> > >    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> > > <__cxxabiv1::__cxa_get_globals()+20>)
> > >    0xf77d01d0 <+12>: add r0, r3
> > >    0xf77d01d2 <+14>: pop {r3, pc}
> > >    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
> > >    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> > > End of assembler dump.
> > 
> > interesting, i thought the linker would relax __tls_get_addr
> > calls in static linked executables to local-exec access model,
> > but maybe that's not implemented on arm.
> > 
> > can you look at the readelf -aW a.out and see if the linker
> > left any dynamic tls relocs in the exe?
> > 
> > > 
> > > OK, so what did __tls_get_addr return?
> > > 
> > > (gdb) b *(__cxa_get_globals + 10)
> > > Breakpoint 3 at 0xf77d01ce: file
> > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> > > (gdb) run
> > > Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> > > 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> > > file or directory.
> > > (gdb) p/x $r0
> > > $3 = 0x1
> > > 
> > > That's doesn't look good either. Let's look at __tls_get_addr.
> > > 
> > > void *__tls_get_addr(tls_mod_off_t *v)
> > > {
> > >         pthread_t self = __pthread_self();
> > >         return (void *)(self->dtv[v[0]] + v[1]);
> > > }
> > > 
> > > (gdb) disas __tls_get_addr (annotations added by me)
> > > Dump of assembler code for function __tls_get_addr:
> > > r2 = v[0]
> > >    0x0000c940 <+0>: ldr r2, [r0, #0]
> > > r3 = tls
> > >    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> > > self = r3 - 120
> > > r3 = *(r3 - 116) = self->dtv
> > >    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> > > r0 = v[1]
> > >    0x0000c94a <+10>: ldr r0, [r0, #4]
> > > r2 = *(r3 + r2 * 4) = dtv[r2]
> > >    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> > > r0 = r0 + r2
> > >    0x0000c950 <+16>: add r0, r2
> > >    0x0000c952 <+18>: bx lr
> > > End of assembler dump.
> > > (gdb) b __tls_get_addr
> > > 
> > > So after pulling out the values I end up with:
> > > v = 0xf77f000c
> > > 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> > > v[0] = 0
> > > v[1] = 0
> > > self = <builtin_tls> 0xf77f064c
> > > 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> > > self->dtv = <builtin_tls+144> 0xf77f06dc
> > > 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> > > return = dtv[v[0]] + v[1] = 1 + 0 = 1
> > > 
> > > At this point I'm out of my depth.
> > 
> > dtv[0] is special and just stores the length of the dtv
> > (which is correctly 1, meaning there is 1 elf module with tls).
> > 
> > so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
> > the tls address of the executable) i think v[0] should be set
> > up by the static linker so it may be a binutils bug.
> > (or maybe static linking is special and then dtv[0] should be
> > set equal to dtv[1]?)
> 
> MIPS had the exact same bug; I think it's fixed upstream now since we
> had the patch for binutils 2.27 in mcm but dropped it in 2.32:
> 
> https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.27/0004-mips-pie-tls.diff
> 
> It affects static PIE only, so -no-pie should be able to work around
> it (conditional for setting the 1 was essentially !ET_DYN instead of
> is_executable).
> 
> We could work around this in musl in one of two ways: rcrt1 could
> process DTPMOD relocs and write a 1, or the static version of
> __init_tls could just duplicate dtv[1] in dtv[0] (the latter is only
> valid up until we add static-linked dlopen, if we ever do that).
> However I was fairly strongly against doing that for MIPS, since it
> was a binutils bug and only affected static linking (so it's not a
> runtime ABI issue; the binary either works at link-time or it doesn't)
> and I'd lean towards treating ARM the same.
> 
> It should be easy to make and apply the binutils patch to fix this,
> and shouldn't be too hard to grep for the same bug in all archs (since
> binutils nicely duplicates this logic for every single arch, for no
> reason whatsoever).

Attached patch is untested but should fix it.

Rich

[-- Attachment #2: 0002-arm-pie-tls.diff --]
[-- Type: text/plain, Size: 364 bytes --]

--- binutils-2.32/bfd/elf32-arm.c.orig	2019-10-02 07:47:36.153918869 -0400
+++ binutils-2.32/bfd/elf32-arm.c	2019-10-02 07:47:45.405988597 -0400
@@ -11624,7 +11624,7 @@
 	  {
 	    /* If we don't know the module number, create a relocation
 	       for it.  */
-	    if (bfd_link_pic (info))
+	    if (bfd_link_dll (info))
 	      {
 		Elf_Internal_Rela outrel;
 

Re: armv7-m segv when throwing c++ exception

[Rich Felker]

On Wed, Oct 02, 2019 at 07:49:55AM -0400, Rich Felker wrote:
> On Wed, Oct 02, 2019 at 07:40:45AM -0400, Rich Felker wrote:
> > On Wed, Oct 02, 2019 at 11:22:33AM +0200, Szabolcs Nagy wrote:
> > > * Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> > > > Hi all,
> > > > 
> > > > I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> > > > using musl-cross-make for armv7-m.
> > > > 
> > > > % cat test.cpp
> > > > int main()
> > > > {
> > > >         throw 0;
> > > > }
> > > > % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> > > > % qemu-arm ./a.out
> > > > qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > > [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> > > > 
> > > > We should be seeing something like "terminate called after throwing an
> > > > instance of 'int'".
> > > 
> > > there is at least one known issue in this area:
> > > libgcc uses pthread apis via weak refs so with static
> > > linking they don't get linked in, however i think that
> > > should only affect unwinding in multi-thread processes,
> > > so this is something else
> > > 
> > > > I have an ODROID-C2 here which is a convenient place to run gdb:
> > > > 
> > > > $ gdb -q ./a.out
> > > > Reading symbols from ./a.out...
> > > > (gdb) run
> > > > Program received signal SIGSEGV, Segmentation fault.
> > > > 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> > > > file or directory.
> > > > (gdb) bt
> > > > #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > #1  0xf77cf902 in main () at test.cpp:3
> > > > (gdb) disas
> > > > Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> > > > std::type_info*, void (*)(void*)):
> > > >    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
> > > >    0xf77cf9e2 <+2>: mov r5, r0
> > > >    0xf77cf9e4 <+4>: mov r7, r1
> > > >    0xf77cf9e6 <+6>: mov r6, r2
> > > >    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
> > > >    0xf77cf9ec <+12>: mov r4, r0
> > > >    0xf77cf9ee <+14>: mov r1, r7
> > > >    0xf77cf9f0 <+16>: mov r2, r6
> > > >    0xf77cf9f2 <+18>: mov r0, r5
> > > > => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
> > > >    0xf77cf9f6 <+22>: adds r3, #1
> > > >    0xf77cf9f8 <+24>: str r3, [r4, #4]
> > > >    0xf77cf9fa <+26>: bl 0xf77cf984
> > > > <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> > > > (*)(void*))>
> > > >    0xf77cf9fe <+30>: movs r3, #1
> > > >    0xf77cfa00 <+32>: mov r4, r0
> > > >    0xf77cfa02 <+34>: str.w r3, [r4], #40
> > > >    0xf77cfa06 <+38>: mov r0, r4
> > > >    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
> > > >    0xf77cfa0c <+44>: mov r0, r4
> > > >    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
> > > >    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> > > > End of assembler dump.
> > > > (gdb) p/x $r4
> > > > $1 = 0x1
> > > > 
> > > > That doesn't look good.
> > > > 
> > > > (gdb) disas __cxa_get_globals
> > > > Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
> > > >    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> > > > <__cxxabiv1::__cxa_get_globals()+16>)
> > > >    0xf77d01c6 <+2>: push {r3, lr}
> > > >    0xf77d01c8 <+4>: add r0, pc
> > > >    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
> > > >    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> > > > <__cxxabiv1::__cxa_get_globals()+20>)
> > > >    0xf77d01d0 <+12>: add r0, r3
> > > >    0xf77d01d2 <+14>: pop {r3, pc}
> > > >    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
> > > >    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> > > > End of assembler dump.
> > > 
> > > interesting, i thought the linker would relax __tls_get_addr
> > > calls in static linked executables to local-exec access model,
> > > but maybe that's not implemented on arm.
> > > 
> > > can you look at the readelf -aW a.out and see if the linker
> > > left any dynamic tls relocs in the exe?
> > > 
> > > > 
> > > > OK, so what did __tls_get_addr return?
> > > > 
> > > > (gdb) b *(__cxa_get_globals + 10)
> > > > Breakpoint 3 at 0xf77d01ce: file
> > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> > > > (gdb) run
> > > > Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> > > > 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> > > > file or directory.
> > > > (gdb) p/x $r0
> > > > $3 = 0x1
> > > > 
> > > > That's doesn't look good either. Let's look at __tls_get_addr.
> > > > 
> > > > void *__tls_get_addr(tls_mod_off_t *v)
> > > > {
> > > >         pthread_t self = __pthread_self();
> > > >         return (void *)(self->dtv[v[0]] + v[1]);
> > > > }
> > > > 
> > > > (gdb) disas __tls_get_addr (annotations added by me)
> > > > Dump of assembler code for function __tls_get_addr:
> > > > r2 = v[0]
> > > >    0x0000c940 <+0>: ldr r2, [r0, #0]
> > > > r3 = tls
> > > >    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> > > > self = r3 - 120
> > > > r3 = *(r3 - 116) = self->dtv
> > > >    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> > > > r0 = v[1]
> > > >    0x0000c94a <+10>: ldr r0, [r0, #4]
> > > > r2 = *(r3 + r2 * 4) = dtv[r2]
> > > >    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> > > > r0 = r0 + r2
> > > >    0x0000c950 <+16>: add r0, r2
> > > >    0x0000c952 <+18>: bx lr
> > > > End of assembler dump.
> > > > (gdb) b __tls_get_addr
> > > > 
> > > > So after pulling out the values I end up with:
> > > > v = 0xf77f000c
> > > > 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> > > > v[0] = 0
> > > > v[1] = 0
> > > > self = <builtin_tls> 0xf77f064c
> > > > 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> > > > self->dtv = <builtin_tls+144> 0xf77f06dc
> > > > 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> > > > return = dtv[v[0]] + v[1] = 1 + 0 = 1
> > > > 
> > > > At this point I'm out of my depth.
> > > 
> > > dtv[0] is special and just stores the length of the dtv
> > > (which is correctly 1, meaning there is 1 elf module with tls).
> > > 
> > > so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
> > > the tls address of the executable) i think v[0] should be set
> > > up by the static linker so it may be a binutils bug.
> > > (or maybe static linking is special and then dtv[0] should be
> > > set equal to dtv[1]?)
> > 
> > MIPS had the exact same bug; I think it's fixed upstream now since we
> > had the patch for binutils 2.27 in mcm but dropped it in 2.32:
> > 
> > https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.27/0004-mips-pie-tls.diff
> > 
> > It affects static PIE only, so -no-pie should be able to work around
> > it (conditional for setting the 1 was essentially !ET_DYN instead of
> > is_executable).
> > 
> > We could work around this in musl in one of two ways: rcrt1 could
> > process DTPMOD relocs and write a 1, or the static version of
> > __init_tls could just duplicate dtv[1] in dtv[0] (the latter is only
> > valid up until we add static-linked dlopen, if we ever do that).
> > However I was fairly strongly against doing that for MIPS, since it
> > was a binutils bug and only affected static linking (so it's not a
> > runtime ABI issue; the binary either works at link-time or it doesn't)
> > and I'd lean towards treating ARM the same.
> > 
> > It should be easy to make and apply the binutils patch to fix this,
> > and shouldn't be too hard to grep for the same bug in all archs (since
> > binutils nicely duplicates this logic for every single arch, for no
> > reason whatsoever).
> 
> Attached patch is untested but should fix it.

It's missing a second instance and doesn't work. I'll send an updated
patch after testing.

Rich

Re: armv7-m segv when throwing c++ exception

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 8627 bytes --]

On Wed, Oct 02, 2019 at 08:25:43AM -0400, Rich Felker wrote:
> On Wed, Oct 02, 2019 at 07:49:55AM -0400, Rich Felker wrote:
> > On Wed, Oct 02, 2019 at 07:40:45AM -0400, Rich Felker wrote:
> > > On Wed, Oct 02, 2019 at 11:22:33AM +0200, Szabolcs Nagy wrote:
> > > > * Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> > > > > Hi all,
> > > > > 
> > > > > I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> > > > > using musl-cross-make for armv7-m.
> > > > > 
> > > > > % cat test.cpp
> > > > > int main()
> > > > > {
> > > > >         throw 0;
> > > > > }
> > > > > % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> > > > > % qemu-arm ./a.out
> > > > > qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > > > [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> > > > > 
> > > > > We should be seeing something like "terminate called after throwing an
> > > > > instance of 'int'".
> > > > 
> > > > there is at least one known issue in this area:
> > > > libgcc uses pthread apis via weak refs so with static
> > > > linking they don't get linked in, however i think that
> > > > should only affect unwinding in multi-thread processes,
> > > > so this is something else
> > > > 
> > > > > I have an ODROID-C2 here which is a convenient place to run gdb:
> > > > > 
> > > > > $ gdb -q ./a.out
> > > > > Reading symbols from ./a.out...
> > > > > (gdb) run
> > > > > Program received signal SIGSEGV, Segmentation fault.
> > > > > 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > > 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> > > > > file or directory.
> > > > > (gdb) bt
> > > > > #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > > #1  0xf77cf902 in main () at test.cpp:3
> > > > > (gdb) disas
> > > > > Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> > > > > std::type_info*, void (*)(void*)):
> > > > >    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
> > > > >    0xf77cf9e2 <+2>: mov r5, r0
> > > > >    0xf77cf9e4 <+4>: mov r7, r1
> > > > >    0xf77cf9e6 <+6>: mov r6, r2
> > > > >    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
> > > > >    0xf77cf9ec <+12>: mov r4, r0
> > > > >    0xf77cf9ee <+14>: mov r1, r7
> > > > >    0xf77cf9f0 <+16>: mov r2, r6
> > > > >    0xf77cf9f2 <+18>: mov r0, r5
> > > > > => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
> > > > >    0xf77cf9f6 <+22>: adds r3, #1
> > > > >    0xf77cf9f8 <+24>: str r3, [r4, #4]
> > > > >    0xf77cf9fa <+26>: bl 0xf77cf984
> > > > > <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> > > > > (*)(void*))>
> > > > >    0xf77cf9fe <+30>: movs r3, #1
> > > > >    0xf77cfa00 <+32>: mov r4, r0
> > > > >    0xf77cfa02 <+34>: str.w r3, [r4], #40
> > > > >    0xf77cfa06 <+38>: mov r0, r4
> > > > >    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
> > > > >    0xf77cfa0c <+44>: mov r0, r4
> > > > >    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
> > > > >    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> > > > > End of assembler dump.
> > > > > (gdb) p/x $r4
> > > > > $1 = 0x1
> > > > > 
> > > > > That doesn't look good.
> > > > > 
> > > > > (gdb) disas __cxa_get_globals
> > > > > Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
> > > > >    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> > > > > <__cxxabiv1::__cxa_get_globals()+16>)
> > > > >    0xf77d01c6 <+2>: push {r3, lr}
> > > > >    0xf77d01c8 <+4>: add r0, pc
> > > > >    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
> > > > >    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> > > > > <__cxxabiv1::__cxa_get_globals()+20>)
> > > > >    0xf77d01d0 <+12>: add r0, r3
> > > > >    0xf77d01d2 <+14>: pop {r3, pc}
> > > > >    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
> > > > >    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> > > > > End of assembler dump.
> > > > 
> > > > interesting, i thought the linker would relax __tls_get_addr
> > > > calls in static linked executables to local-exec access model,
> > > > but maybe that's not implemented on arm.
> > > > 
> > > > can you look at the readelf -aW a.out and see if the linker
> > > > left any dynamic tls relocs in the exe?
> > > > 
> > > > > 
> > > > > OK, so what did __tls_get_addr return?
> > > > > 
> > > > > (gdb) b *(__cxa_get_globals + 10)
> > > > > Breakpoint 3 at 0xf77d01ce: file
> > > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> > > > > (gdb) run
> > > > > Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> > > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> > > > > 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> > > > > file or directory.
> > > > > (gdb) p/x $r0
> > > > > $3 = 0x1
> > > > > 
> > > > > That's doesn't look good either. Let's look at __tls_get_addr.
> > > > > 
> > > > > void *__tls_get_addr(tls_mod_off_t *v)
> > > > > {
> > > > >         pthread_t self = __pthread_self();
> > > > >         return (void *)(self->dtv[v[0]] + v[1]);
> > > > > }
> > > > > 
> > > > > (gdb) disas __tls_get_addr (annotations added by me)
> > > > > Dump of assembler code for function __tls_get_addr:
> > > > > r2 = v[0]
> > > > >    0x0000c940 <+0>: ldr r2, [r0, #0]
> > > > > r3 = tls
> > > > >    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> > > > > self = r3 - 120
> > > > > r3 = *(r3 - 116) = self->dtv
> > > > >    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> > > > > r0 = v[1]
> > > > >    0x0000c94a <+10>: ldr r0, [r0, #4]
> > > > > r2 = *(r3 + r2 * 4) = dtv[r2]
> > > > >    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> > > > > r0 = r0 + r2
> > > > >    0x0000c950 <+16>: add r0, r2
> > > > >    0x0000c952 <+18>: bx lr
> > > > > End of assembler dump.
> > > > > (gdb) b __tls_get_addr
> > > > > 
> > > > > So after pulling out the values I end up with:
> > > > > v = 0xf77f000c
> > > > > 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> > > > > v[0] = 0
> > > > > v[1] = 0
> > > > > self = <builtin_tls> 0xf77f064c
> > > > > 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> > > > > self->dtv = <builtin_tls+144> 0xf77f06dc
> > > > > 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> > > > > return = dtv[v[0]] + v[1] = 1 + 0 = 1
> > > > > 
> > > > > At this point I'm out of my depth.
> > > > 
> > > > dtv[0] is special and just stores the length of the dtv
> > > > (which is correctly 1, meaning there is 1 elf module with tls).
> > > > 
> > > > so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
> > > > the tls address of the executable) i think v[0] should be set
> > > > up by the static linker so it may be a binutils bug.
> > > > (or maybe static linking is special and then dtv[0] should be
> > > > set equal to dtv[1]?)
> > > 
> > > MIPS had the exact same bug; I think it's fixed upstream now since we
> > > had the patch for binutils 2.27 in mcm but dropped it in 2.32:
> > > 
> > > https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.27/0004-mips-pie-tls.diff
> > > 
> > > It affects static PIE only, so -no-pie should be able to work around
> > > it (conditional for setting the 1 was essentially !ET_DYN instead of
> > > is_executable).
> > > 
> > > We could work around this in musl in one of two ways: rcrt1 could
> > > process DTPMOD relocs and write a 1, or the static version of
> > > __init_tls could just duplicate dtv[1] in dtv[0] (the latter is only
> > > valid up until we add static-linked dlopen, if we ever do that).
> > > However I was fairly strongly against doing that for MIPS, since it
> > > was a binutils bug and only affected static linking (so it's not a
> > > runtime ABI issue; the binary either works at link-time or it doesn't)
> > > and I'd lean towards treating ARM the same.
> > > 
> > > It should be easy to make and apply the binutils patch to fix this,
> > > and shouldn't be too hard to grep for the same bug in all archs (since
> > > binutils nicely duplicates this logic for every single arch, for no
> > > reason whatsoever).
> > 
> > Attached patch is untested but should fix it.
> 
> It's missing a second instance and doesn't work. I'll send an updated
> patch after testing.

Attached works for me.

Rich

[-- Attachment #2: 0002-arm-pie-tls.diff --]
[-- Type: text/plain, Size: 700 bytes --]

--- binutils-2.32/bfd/elf32-arm.c.orig	2019-10-02 07:47:36.153918869 -0400
+++ binutils-2.32/bfd/elf32-arm.c	2019-10-02 08:37:09.108263016 -0400
@@ -11624,7 +11624,7 @@
 	  {
 	    /* If we don't know the module number, create a relocation
 	       for it.  */
-	    if (bfd_link_pic (info))
+	    if (bfd_link_dll (info))
 	      {
 		Elf_Internal_Rela outrel;
 
@@ -11728,7 +11728,7 @@
 	       now, and emit any relocations.  If both an IE GOT and a
 	       GD GOT are necessary, we emit the GD first.  */
 
-	    if ((bfd_link_pic (info) || indx != 0)
+	    if ((bfd_link_dll (info) || indx != 0)
 		&& (h == NULL
 		    || (ELF_ST_VISIBILITY (h->other) == STV_DEFAULT
 			&& !resolved_to_zero)

Re: armv7-m segv when throwing c++ exception

[Rich Felker]

On Wed, Oct 02, 2019 at 08:49:12AM -0400, Rich Felker wrote:
> On Wed, Oct 02, 2019 at 08:25:43AM -0400, Rich Felker wrote:
> > On Wed, Oct 02, 2019 at 07:49:55AM -0400, Rich Felker wrote:
> > > On Wed, Oct 02, 2019 at 07:40:45AM -0400, Rich Felker wrote:
> > > > On Wed, Oct 02, 2019 at 11:22:33AM +0200, Szabolcs Nagy wrote:
> > > > > * Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> > > > > > Hi all,
> > > > > > 
> > > > > > I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> > > > > > using musl-cross-make for armv7-m.
> > > > > > 
> > > > > > % cat test.cpp
> > > > > > int main()
> > > > > > {
> > > > > >         throw 0;
> > > > > > }
> > > > > > % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> > > > > > % qemu-arm ./a.out
> > > > > > qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > > > > [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> > > > > > 
> > > > > > We should be seeing something like "terminate called after throwing an
> > > > > > instance of 'int'".
> > > > > 
> > > > > there is at least one known issue in this area:
> > > > > libgcc uses pthread apis via weak refs so with static
> > > > > linking they don't get linked in, however i think that
> > > > > should only affect unwinding in multi-thread processes,
> > > > > so this is something else
> > > > > 
> > > > > > I have an ODROID-C2 here which is a convenient place to run gdb:
> > > > > > 
> > > > > > $ gdb -q ./a.out
> > > > > > Reading symbols from ./a.out...
> > > > > > (gdb) run
> > > > > > Program received signal SIGSEGV, Segmentation fault.
> > > > > > 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > > > 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> > > > > > file or directory.
> > > > > > (gdb) bt
> > > > > > #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > > > #1  0xf77cf902 in main () at test.cpp:3
> > > > > > (gdb) disas
> > > > > > Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> > > > > > std::type_info*, void (*)(void*)):
> > > > > >    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
> > > > > >    0xf77cf9e2 <+2>: mov r5, r0
> > > > > >    0xf77cf9e4 <+4>: mov r7, r1
> > > > > >    0xf77cf9e6 <+6>: mov r6, r2
> > > > > >    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
> > > > > >    0xf77cf9ec <+12>: mov r4, r0
> > > > > >    0xf77cf9ee <+14>: mov r1, r7
> > > > > >    0xf77cf9f0 <+16>: mov r2, r6
> > > > > >    0xf77cf9f2 <+18>: mov r0, r5
> > > > > > => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
> > > > > >    0xf77cf9f6 <+22>: adds r3, #1
> > > > > >    0xf77cf9f8 <+24>: str r3, [r4, #4]
> > > > > >    0xf77cf9fa <+26>: bl 0xf77cf984
> > > > > > <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> > > > > > (*)(void*))>
> > > > > >    0xf77cf9fe <+30>: movs r3, #1
> > > > > >    0xf77cfa00 <+32>: mov r4, r0
> > > > > >    0xf77cfa02 <+34>: str.w r3, [r4], #40
> > > > > >    0xf77cfa06 <+38>: mov r0, r4
> > > > > >    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
> > > > > >    0xf77cfa0c <+44>: mov r0, r4
> > > > > >    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
> > > > > >    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> > > > > > End of assembler dump.
> > > > > > (gdb) p/x $r4
> > > > > > $1 = 0x1
> > > > > > 
> > > > > > That doesn't look good.
> > > > > > 
> > > > > > (gdb) disas __cxa_get_globals
> > > > > > Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
> > > > > >    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> > > > > > <__cxxabiv1::__cxa_get_globals()+16>)
> > > > > >    0xf77d01c6 <+2>: push {r3, lr}
> > > > > >    0xf77d01c8 <+4>: add r0, pc
> > > > > >    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
> > > > > >    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> > > > > > <__cxxabiv1::__cxa_get_globals()+20>)
> > > > > >    0xf77d01d0 <+12>: add r0, r3
> > > > > >    0xf77d01d2 <+14>: pop {r3, pc}
> > > > > >    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
> > > > > >    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> > > > > > End of assembler dump.
> > > > > 
> > > > > interesting, i thought the linker would relax __tls_get_addr
> > > > > calls in static linked executables to local-exec access model,
> > > > > but maybe that's not implemented on arm.
> > > > > 
> > > > > can you look at the readelf -aW a.out and see if the linker
> > > > > left any dynamic tls relocs in the exe?
> > > > > 
> > > > > > 
> > > > > > OK, so what did __tls_get_addr return?
> > > > > > 
> > > > > > (gdb) b *(__cxa_get_globals + 10)
> > > > > > Breakpoint 3 at 0xf77d01ce: file
> > > > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> > > > > > (gdb) run
> > > > > > Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> > > > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> > > > > > 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> > > > > > file or directory.
> > > > > > (gdb) p/x $r0
> > > > > > $3 = 0x1
> > > > > > 
> > > > > > That's doesn't look good either. Let's look at __tls_get_addr.
> > > > > > 
> > > > > > void *__tls_get_addr(tls_mod_off_t *v)
> > > > > > {
> > > > > >         pthread_t self = __pthread_self();
> > > > > >         return (void *)(self->dtv[v[0]] + v[1]);
> > > > > > }
> > > > > > 
> > > > > > (gdb) disas __tls_get_addr (annotations added by me)
> > > > > > Dump of assembler code for function __tls_get_addr:
> > > > > > r2 = v[0]
> > > > > >    0x0000c940 <+0>: ldr r2, [r0, #0]
> > > > > > r3 = tls
> > > > > >    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> > > > > > self = r3 - 120
> > > > > > r3 = *(r3 - 116) = self->dtv
> > > > > >    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> > > > > > r0 = v[1]
> > > > > >    0x0000c94a <+10>: ldr r0, [r0, #4]
> > > > > > r2 = *(r3 + r2 * 4) = dtv[r2]
> > > > > >    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> > > > > > r0 = r0 + r2
> > > > > >    0x0000c950 <+16>: add r0, r2
> > > > > >    0x0000c952 <+18>: bx lr
> > > > > > End of assembler dump.
> > > > > > (gdb) b __tls_get_addr
> > > > > > 
> > > > > > So after pulling out the values I end up with:
> > > > > > v = 0xf77f000c
> > > > > > 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> > > > > > v[0] = 0
> > > > > > v[1] = 0
> > > > > > self = <builtin_tls> 0xf77f064c
> > > > > > 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> > > > > > self->dtv = <builtin_tls+144> 0xf77f06dc
> > > > > > 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> > > > > > return = dtv[v[0]] + v[1] = 1 + 0 = 1
> > > > > > 
> > > > > > At this point I'm out of my depth.
> > > > > 
> > > > > dtv[0] is special and just stores the length of the dtv
> > > > > (which is correctly 1, meaning there is 1 elf module with tls).
> > > > > 
> > > > > so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
> > > > > the tls address of the executable) i think v[0] should be set
> > > > > up by the static linker so it may be a binutils bug.
> > > > > (or maybe static linking is special and then dtv[0] should be
> > > > > set equal to dtv[1]?)
> > > > 
> > > > MIPS had the exact same bug; I think it's fixed upstream now since we
> > > > had the patch for binutils 2.27 in mcm but dropped it in 2.32:
> > > > 
> > > > https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.27/0004-mips-pie-tls.diff
> > > > 
> > > > It affects static PIE only, so -no-pie should be able to work around
> > > > it (conditional for setting the 1 was essentially !ET_DYN instead of
> > > > is_executable).
> > > > 
> > > > We could work around this in musl in one of two ways: rcrt1 could
> > > > process DTPMOD relocs and write a 1, or the static version of
> > > > __init_tls could just duplicate dtv[1] in dtv[0] (the latter is only
> > > > valid up until we add static-linked dlopen, if we ever do that).
> > > > However I was fairly strongly against doing that for MIPS, since it
> > > > was a binutils bug and only affected static linking (so it's not a
> > > > runtime ABI issue; the binary either works at link-time or it doesn't)
> > > > and I'd lean towards treating ARM the same.
> > > > 
> > > > It should be easy to make and apply the binutils patch to fix this,
> > > > and shouldn't be too hard to grep for the same bug in all archs (since
> > > > binutils nicely duplicates this logic for every single arch, for no
> > > > reason whatsoever).
> > > 
> > > Attached patch is untested but should fix it.
> > 
> > It's missing a second instance and doesn't work. I'll send an updated
> > patch after testing.
> 
> Attached works for me.

> --- binutils-2.32/bfd/elf32-arm.c.orig	2019-10-02 07:47:36.153918869 -0400
> +++ binutils-2.32/bfd/elf32-arm.c	2019-10-02 08:37:09.108263016 -0400
> @@ -11624,7 +11624,7 @@
>  	  {
>  	    /* If we don't know the module number, create a relocation
>  	       for it.  */
> -	    if (bfd_link_pic (info))
> +	    if (bfd_link_dll (info))
>  	      {
>  		Elf_Internal_Rela outrel;
>  
> @@ -11728,7 +11728,7 @@
>  	       now, and emit any relocations.  If both an IE GOT and a
>  	       GD GOT are necessary, we emit the GD first.  */
>  
> -	    if ((bfd_link_pic (info) || indx != 0)
> +	    if ((bfd_link_dll (info) || indx != 0)
>  		&& (h == NULL
>  		    || (ELF_ST_VISIBILITY (h->other) == STV_DEFAULT
>  			&& !resolved_to_zero)

And, filed bug report with patch:

https://sourceware.org/bugzilla/show_bug.cgi?id=25056

Patches are now in musl-cross-make too.

Rich

Re: armv7-m segv when throwing c++ exception

[Patrick Oppenlander]

Wow, that was fast! Thank you!

I really appreciate the effort you & the whole musl community put in.

Patrick

On Thu, Oct 3, 2019 at 12:56 AM Rich Felker <dalias@libc.org> wrote:
>
> On Wed, Oct 02, 2019 at 08:49:12AM -0400, Rich Felker wrote:
> > On Wed, Oct 02, 2019 at 08:25:43AM -0400, Rich Felker wrote:
> > > On Wed, Oct 02, 2019 at 07:49:55AM -0400, Rich Felker wrote:
> > > > On Wed, Oct 02, 2019 at 07:40:45AM -0400, Rich Felker wrote:
> > > > > On Wed, Oct 02, 2019 at 11:22:33AM +0200, Szabolcs Nagy wrote:
> > > > > > * Patrick Oppenlander <patrick.oppenlander@gmail.com> [2019-10-02 12:00:19 +1000]:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I'm running into an issue with a gcc-8.3.0/musl-1.1.23 toolchain built
> > > > > > > using musl-cross-make for armv7-m.
> > > > > > >
> > > > > > > % cat test.cpp
> > > > > > > int main()
> > > > > > > {
> > > > > > >         throw 0;
> > > > > > > }
> > > > > > > % armv7m-linux-musleabihf-g++ -ggdb -static test.cpp
> > > > > > > % qemu-arm ./a.out
> > > > > > > qemu: uncaught target signal 11 (Segmentation fault) - core dumped
> > > > > > > [2]    198246 segmentation fault (core dumped)  qemu-arm ./a.out
> > > > > > >
> > > > > > > We should be seeing something like "terminate called after throwing an
> > > > > > > instance of 'int'".
> > > > > >
> > > > > > there is at least one known issue in this area:
> > > > > > libgcc uses pthread apis via weak refs so with static
> > > > > > linking they don't get linked in, however i think that
> > > > > > should only affect unwinding in multi-thread processes,
> > > > > > so this is something else
> > > > > >
> > > > > > > I have an ODROID-C2 here which is a convenient place to run gdb:
> > > > > > >
> > > > > > > $ gdb -q ./a.out
> > > > > > > Reading symbols from ./a.out...
> > > > > > > (gdb) run
> > > > > > > Program received signal SIGSEGV, Segmentation fault.
> > > > > > > 0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > > > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > > > > 81 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc: No such
> > > > > > > file or directory.
> > > > > > > (gdb) bt
> > > > > > > #0  0xf77cf9f4 in __cxxabiv1::__cxa_throw (obj=0xf77f5fa0,
> > > > > > > tinfo=0xf77efb20 <typeinfo for int>, dest=0x0)
> > > > > > >     at ../../../../src_gcc/libstdc++-v3/libsupc++/eh_throw.cc:81
> > > > > > > #1  0xf77cf902 in main () at test.cpp:3
> > > > > > > (gdb) disas
> > > > > > > Dump of assembler code for function __cxxabiv1::__cxa_throw(void*,
> > > > > > > std::type_info*, void (*)(void*)):
> > > > > > >    0xf77cf9e0 <+0>: push {r3, r4, r5, r6, r7, lr}
> > > > > > >    0xf77cf9e2 <+2>: mov r5, r0
> > > > > > >    0xf77cf9e4 <+4>: mov r7, r1
> > > > > > >    0xf77cf9e6 <+6>: mov r6, r2
> > > > > > >    0xf77cf9e8 <+8>: bl 0xf77d01c4 <__cxxabiv1::__cxa_get_globals()>
> > > > > > >    0xf77cf9ec <+12>: mov r4, r0
> > > > > > >    0xf77cf9ee <+14>: mov r1, r7
> > > > > > >    0xf77cf9f0 <+16>: mov r2, r6
> > > > > > >    0xf77cf9f2 <+18>: mov r0, r5
> > > > > > > => 0xf77cf9f4 <+20>: ldr r3, [r4, #4]
> > > > > > >    0xf77cf9f6 <+22>: adds r3, #1
> > > > > > >    0xf77cf9f8 <+24>: str r3, [r4, #4]
> > > > > > >    0xf77cf9fa <+26>: bl 0xf77cf984
> > > > > > > <__cxxabiv1::__cxa_init_primary_exception(void*, std::type_info*, void
> > > > > > > (*)(void*))>
> > > > > > >    0xf77cf9fe <+30>: movs r3, #1
> > > > > > >    0xf77cfa00 <+32>: mov r4, r0
> > > > > > >    0xf77cfa02 <+34>: str.w r3, [r4], #40
> > > > > > >    0xf77cfa06 <+38>: mov r0, r4
> > > > > > >    0xf77cfa08 <+40>: bl 0xf77d88a8 <___Unwind_RaiseException>
> > > > > > >    0xf77cfa0c <+44>: mov r0, r4
> > > > > > >    0xf77cfa0e <+46>: bl 0xf77d01e0 <__cxxabiv1::__cxa_begin_catch(void*)>
> > > > > > >    0xf77cfa12 <+50>: bl 0xf77cfaf0 <std::terminate()>
> > > > > > > End of assembler dump.
> > > > > > > (gdb) p/x $r4
> > > > > > > $1 = 0x1
> > > > > > >
> > > > > > > That doesn't look good.
> > > > > > >
> > > > > > > (gdb) disas __cxa_get_globals
> > > > > > > Dump of assembler code for function __cxxabiv1::__cxa_get_globals():
> > > > > > >    0xf77d01c4 <+0>: ldr r0, [pc, #12] ; (0xf77d01d4
> > > > > > > <__cxxabiv1::__cxa_get_globals()+16>)
> > > > > > >    0xf77d01c6 <+2>: push {r3, lr}
> > > > > > >    0xf77d01c8 <+4>: add r0, pc
> > > > > > >    0xf77d01ca <+6>: bl 0xf77da940 <__tls_get_addr>
> > > > > > >    0xf77d01ce <+10>: ldr r3, [pc, #8] ; (0xf77d01d8
> > > > > > > <__cxxabiv1::__cxa_get_globals()+20>)
> > > > > > >    0xf77d01d0 <+12>: add r0, r3
> > > > > > >    0xf77d01d2 <+14>: pop {r3, pc}
> > > > > > >    0xf77d01d4 <+16>: ; <UNDEFINED> instruction: 0001fe40
> > > > > > >    0xf77d01d8 <+20>: ; <UNDEFINED> instruction: 00000000
> > > > > > > End of assembler dump.
> > > > > >
> > > > > > interesting, i thought the linker would relax __tls_get_addr
> > > > > > calls in static linked executables to local-exec access model,
> > > > > > but maybe that's not implemented on arm.
> > > > > >
> > > > > > can you look at the readelf -aW a.out and see if the linker
> > > > > > left any dynamic tls relocs in the exe?
> > > > > >
> > > > > > >
> > > > > > > OK, so what did __tls_get_addr return?
> > > > > > >
> > > > > > > (gdb) b *(__cxa_get_globals + 10)
> > > > > > > Breakpoint 3 at 0xf77d01ce: file
> > > > > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc, line 62.
> > > > > > > (gdb) run
> > > > > > > Breakpoint 3, 0xf77d01ce in __cxxabiv1::__cxa_get_globals () at
> > > > > > > ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc:62
> > > > > > > 62 ../../../../src_gcc/libstdc++-v3/libsupc++/eh_globals.cc: No such
> > > > > > > file or directory.
> > > > > > > (gdb) p/x $r0
> > > > > > > $3 = 0x1
> > > > > > >
> > > > > > > That's doesn't look good either. Let's look at __tls_get_addr.
> > > > > > >
> > > > > > > void *__tls_get_addr(tls_mod_off_t *v)
> > > > > > > {
> > > > > > >         pthread_t self = __pthread_self();
> > > > > > >         return (void *)(self->dtv[v[0]] + v[1]);
> > > > > > > }
> > > > > > >
> > > > > > > (gdb) disas __tls_get_addr (annotations added by me)
> > > > > > > Dump of assembler code for function __tls_get_addr:
> > > > > > > r2 = v[0]
> > > > > > >    0x0000c940 <+0>: ldr r2, [r0, #0]
> > > > > > > r3 = tls
> > > > > > >    0x0000c942 <+2>: ; <UNDEFINED> instruction: 0xee1d3f70
> > > > > > > self = r3 - 120
> > > > > > > r3 = *(r3 - 116) = self->dtv
> > > > > > >    0x0000c946 <+6>: ldr.w r3, [r3, #-116]
> > > > > > > r0 = v[1]
> > > > > > >    0x0000c94a <+10>: ldr r0, [r0, #4]
> > > > > > > r2 = *(r3 + r2 * 4) = dtv[r2]
> > > > > > >    0x0000c94c <+12>: ldr.w r2, [r3, r2, lsl #2]
> > > > > > > r0 = r0 + r2
> > > > > > >    0x0000c950 <+16>: add r0, r2
> > > > > > >    0x0000c952 <+18>: bx lr
> > > > > > > End of assembler dump.
> > > > > > > (gdb) b __tls_get_addr
> > > > > > >
> > > > > > > So after pulling out the values I end up with:
> > > > > > > v = 0xf77f000c
> > > > > > > 0xf77f000c: 0x00000000 0x00000000 0xf77f072c 0xf77eff1c
> > > > > > > v[0] = 0
> > > > > > > v[1] = 0
> > > > > > > self = <builtin_tls> 0xf77f064c
> > > > > > > 0xf77f064c <builtin_tls>: 0xf77f064c 0xf77f06dc 0xf77f064c 0xf77f064c
> > > > > > > self->dtv = <builtin_tls+144> 0xf77f06dc
> > > > > > > 0xf77f06dc <builtin_tls+144>: 0x00000001 0xf77f06cc 0x00000000 0x00000000
> > > > > > > return = dtv[v[0]] + v[1] = 1 + 0 = 1
> > > > > > >
> > > > > > > At this point I'm out of my depth.
> > > > > >
> > > > > > dtv[0] is special and just stores the length of the dtv
> > > > > > (which is correctly 1, meaning there is 1 elf module with tls).
> > > > > >
> > > > > > so the issue is that v[0]==0 instead of 1 (dtv[1] would hold
> > > > > > the tls address of the executable) i think v[0] should be set
> > > > > > up by the static linker so it may be a binutils bug.
> > > > > > (or maybe static linking is special and then dtv[0] should be
> > > > > > set equal to dtv[1]?)
> > > > >
> > > > > MIPS had the exact same bug; I think it's fixed upstream now since we
> > > > > had the patch for binutils 2.27 in mcm but dropped it in 2.32:
> > > > >
> > > > > https://github.com/richfelker/musl-cross-make/blob/master/patches/binutils-2.27/0004-mips-pie-tls.diff
> > > > >
> > > > > It affects static PIE only, so -no-pie should be able to work around
> > > > > it (conditional for setting the 1 was essentially !ET_DYN instead of
> > > > > is_executable).
> > > > >
> > > > > We could work around this in musl in one of two ways: rcrt1 could
> > > > > process DTPMOD relocs and write a 1, or the static version of
> > > > > __init_tls could just duplicate dtv[1] in dtv[0] (the latter is only
> > > > > valid up until we add static-linked dlopen, if we ever do that).
> > > > > However I was fairly strongly against doing that for MIPS, since it
> > > > > was a binutils bug and only affected static linking (so it's not a
> > > > > runtime ABI issue; the binary either works at link-time or it doesn't)
> > > > > and I'd lean towards treating ARM the same.
> > > > >
> > > > > It should be easy to make and apply the binutils patch to fix this,
> > > > > and shouldn't be too hard to grep for the same bug in all archs (since
> > > > > binutils nicely duplicates this logic for every single arch, for no
> > > > > reason whatsoever).
> > > >
> > > > Attached patch is untested but should fix it.
> > >
> > > It's missing a second instance and doesn't work. I'll send an updated
> > > patch after testing.
> >
> > Attached works for me.
>
> > --- binutils-2.32/bfd/elf32-arm.c.orig        2019-10-02 07:47:36.153918869 -0400
> > +++ binutils-2.32/bfd/elf32-arm.c     2019-10-02 08:37:09.108263016 -0400
> > @@ -11624,7 +11624,7 @@
> >         {
> >           /* If we don't know the module number, create a relocation
> >              for it.  */
> > -         if (bfd_link_pic (info))
> > +         if (bfd_link_dll (info))
> >             {
> >               Elf_Internal_Rela outrel;
> >
> > @@ -11728,7 +11728,7 @@
> >              now, and emit any relocations.  If both an IE GOT and a
> >              GD GOT are necessary, we emit the GD first.  */
> >
> > -         if ((bfd_link_pic (info) || indx != 0)
> > +         if ((bfd_link_dll (info) || indx != 0)
> >               && (h == NULL
> >                   || (ELF_ST_VISIBILITY (h->other) == STV_DEFAULT
> >                       && !resolved_to_zero)
>
> And, filed bug report with patch:
>
> https://sourceware.org/bugzilla/show_bug.cgi?id=25056
>
> Patches are now in musl-cross-make too.
>
> Rich