[musl] incomplete GDB backtrace stacks with MUSL cross toolchain

[musl] incomplete GDB backtrace stacks with MUSL cross toolchain

[Colen Garoutte-Carson]

[-- Attachment #1: Type: text/plain, Size: 1731 bytes --]

Hi,

                 We're using MUSL cross-compilation toolchains to build fully statically linked binaries, targeting x86_64, aarch64, and armel. (Which allows us to work around some issues with glibc incompatibilities)

                 If we connect GDB to our process and get a backtrace, we get very little.  i.e.:

__syscall_cp_c (Unknown Source:0)
__timedwait_cp (Unknown Source:0)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)

Or:

__stdio_read (Unknown Source:0)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)

                 We'd really like to get the rest of these stacks to investigate issues such as deadlocks, for example (among several other reasons, such as our own crash reporting handler).

                 I found related info on StackOverflow, such as: debugging - Why does gdb backtrace show only one frame when catching syscall? - Stack Overflow<https://stackoverflow.com/questions/29764951/why-does-gdb-backtrace-show-only-one-frame-when-catching-syscall>
                 And: c - Stack frame NULL in backtrace log - Stack Overflow<https://stackoverflow.com/questions/26214936/stack-frame-null-in-backtrace-log>

                 My hypothesis is that we're calling into code that does not provide the stack unwind descriptors needed by GDB.  The first of the above links refers specifically to MUSL and patching it (but for MIPS).

                 I've tried building a fresh MUSL (1.2.4) GCC (13.2.0) toolchain, and the issue persists with a binary built with it.

                 Can someone explain what we're running into here, and whether there is a potential solution (even if we may have to contribute it ourselves)?

Thanks!,

  *   Colen


[-- Attachment #2: Type: text/html, Size: 10100 bytes --]

Re: [musl] incomplete GDB backtrace stacks with MUSL cross toolchain

[Szabolcs Nagy]

* Colen Garoutte-Carson <coleng@microsoft.com> [2024-03-13 01:16:47 +0000]:
> Hi,
> 
>                  We're using MUSL cross-compilation toolchains to build fully statically linked binaries, targeting x86_64, aarch64, and armel. (Which allows us to work around some issues with glibc incompatibilities)
> 
>                  If we connect GDB to our process and get a backtrace, we get very little.  i.e.:
> 
> __syscall_cp_c (Unknown Source:0)
> __timedwait_cp (Unknown Source:0)
> [Unknown/Just-In-Time compiled code] (Unknown Source:0)
> 
> Or:
> 
> __stdio_read (Unknown Source:0)
> [Unknown/Just-In-Time compiled code] (Unknown Source:0)
> 
>                  We'd really like to get the rest of these stacks to investigate issues such as deadlocks, for example (among several other reasons, such as our own crash reporting handler).
> 
>                  I found related info on StackOverflow, such as: debugging - Why does gdb backtrace show only one frame when catching syscall? - Stack Overflow<https://stackoverflow.com/questions/29764951/why-does-gdb-backtrace-show-only-one-frame-when-catching-syscall>
>                  And: c - Stack frame NULL in backtrace log - Stack Overflow<https://stackoverflow.com/questions/26214936/stack-frame-null-in-backtrace-log>
> 
>                  My hypothesis is that we're calling into code that does not provide the stack unwind descriptors needed by GDB.  The first of the above links refers specifically to MUSL and patching it (but for MIPS).
> 
>                  I've tried building a fresh MUSL (1.2.4) GCC (13.2.0) toolchain, and the issue persists with a binary built with it.
> 
>                  Can someone explain what we're running into here, and whether there is a potential solution (even if we may have to contribute it ourselves)?
> 

the first thing to note is that --enable-debug is not the default,
so make sure you configure musl with that when building the cross
toolchain. (e.g. if you use musl-cross-make then you need
MUSL_CONFIG += --enable-debug in your config.mak)

the second thing to note is that musl only has debug info for asm
files on x86 targets (the cfi directives are generated by a script,
tools/add-cfi.*.awk, if you provide that for another target that
will work too), this may affect backtrace from __syscall_cp_asm
(although i expect gdb to figure this one out without cfi, since
gdb should use heuristics when cfi is not available).

there can be subtle issues with unwinding across signal handlers,
but otherwise backtrace across c code should work if your compiler
emits debug info (gcc -g).

> Thanks!,
> 
>   *   Colen
>