Re: [musl] Unwinding multithreaded musl applications with elfutils fails

[Szabolcs Nagy <nsz@port70.net>]

* Matt Wozniski <godlygeek@gmail.com> [2023-04-01 22:57:09 -0400]:
> On Fri, Mar 31, 2023 at 7:40 AM Szabolcs Nagy <nsz@port70.net> wrote:
> >
> > * Matt Wozniski <godlygeek@gmail.com> [2023-03-30 22:43:28 -0400]:
> > > Using the elfutils eu-stack program or libdw's dwfl_getthread_frames
> > > API to unwind multithreaded applications linked against musl libc on
> > > x86-64 fails, getting stuck on `__clone`:
> >
> > musl has limited cfi debug info support (target specific), likely the
> > unwinder needs a
> >
> >   .cfi_undefined rip
> >
> > in the clone start function to know where the stack frames end.
> ...
> > musl supports building things without any cfi debug info since c
> > does not require unwind support, but linux systems nowadays assume
> > unwind tables are part of the platform abi so musl based distros
> > should probably include it.
> ...
> > musl does not guarantee frame-pointers either
> 
> So, if I understand what you're saying correctly: musl itself doesn't
> guarantee the ability to unwind through it at all (neither using DWARF
> unwind tables nor using frame pointers), but musl based distros like
> Alpine ought to include proper unwind tables. Does that mean that you
> don't consider the lack of CFI for `__clone` a defect in musl, but
> that it's still worth reporting to the Alpine musl maintainers as a
> defect in Alpine's musl build?
> 
> If so, what would distro maintainers have to do in order to remedy
> that defect? Would it be patches to the (target specific) `clone.s` to
> add appropriate CFI when building musl for the distro?

musl has no cfi annotation by default, but there is a tool that adds
it to asm on some targets and the compiler can generate cfi for c code.

i think distros should enable cfi when building musl (currently it is
only in debug builds i think).

but it seems this is not enough to mark the end of the stack frames.

> > (it could figure out the end with the same heuristic that gdb uses,
> > but apparently elfutils is not smart enough).
> >
> > some backtracers may want cleared frame-pointer (rbp=0) to detect
> > the end.
> ...
> > rbp=0 may be the reason why backtrace in the main thread works, so it
> > may be enough to do that in threads too.
> 
> And it sounds like both of these are workarounds that elfutils might
> be able to pursue in the absence of correct unwind information built
> into musl itself. Thanks, that gives a useful direction to dig in.

it seems __clone already has xor %ebp,%ebp

maybe we need a rule in add-cfi.x86_64.awk to emit cfi based on that.