From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.1 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 Received: from second.openwall.net (second.openwall.net [193.110.157.125]) by inbox.vuxu.org (Postfix) with SMTP id 3D08226800 for ; Wed, 13 Mar 2024 21:02:09 +0100 (CET) Received: (qmail 28295 invoked by uid 550); 13 Mar 2024 19:57:53 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Reply-To: musl@lists.openwall.com Received: (qmail 28254 invoked from network); 13 Mar 2024 19:57:53 -0000 Date: Wed, 13 Mar 2024 21:01:54 +0100 From: Szabolcs Nagy To: Colen Garoutte-Carson Cc: "musl@lists.openwall.com" Message-ID: <20240313200154.GH1884416@port70.net> Mail-Followup-To: Colen Garoutte-Carson , "musl@lists.openwall.com" References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [musl] incomplete GDB backtrace stacks with MUSL cross toolchain * Colen Garoutte-Carson [2024-03-13 01:16:47 +0000]: > Hi, > > We're using MUSL cross-compilation toolchains to build fully statically linked binaries, targeting x86_64, aarch64, and armel. (Which allows us to work around some issues with glibc incompatibilities) > > If we connect GDB to our process and get a backtrace, we get very little. i.e.: > > __syscall_cp_c (Unknown Source:0) > __timedwait_cp (Unknown Source:0) > [Unknown/Just-In-Time compiled code] (Unknown Source:0) > > Or: > > __stdio_read (Unknown Source:0) > [Unknown/Just-In-Time compiled code] (Unknown Source:0) > > We'd really like to get the rest of these stacks to investigate issues such as deadlocks, for example (among several other reasons, such as our own crash reporting handler). > > I found related info on StackOverflow, such as: debugging - Why does gdb backtrace show only one frame when catching syscall? - Stack Overflow > And: c - Stack frame NULL in backtrace log - Stack Overflow > > My hypothesis is that we're calling into code that does not provide the stack unwind descriptors needed by GDB. The first of the above links refers specifically to MUSL and patching it (but for MIPS). > > I've tried building a fresh MUSL (1.2.4) GCC (13.2.0) toolchain, and the issue persists with a binary built with it. > > Can someone explain what we're running into here, and whether there is a potential solution (even if we may have to contribute it ourselves)? > the first thing to note is that --enable-debug is not the default, so make sure you configure musl with that when building the cross toolchain. (e.g. if you use musl-cross-make then you need MUSL_CONFIG += --enable-debug in your config.mak) the second thing to note is that musl only has debug info for asm files on x86 targets (the cfi directives are generated by a script, tools/add-cfi.*.awk, if you provide that for another target that will work too), this may affect backtrace from __syscall_cp_asm (although i expect gdb to figure this one out without cfi, since gdb should use heuristics when cfi is not available). there can be subtle issues with unwinding across signal handlers, but otherwise backtrace across c code should work if your compiler emits debug info (gcc -g). > Thanks!, > > * Colen >