[musl] [PATCH] x86_64: Generate an eh-frame entry for syscall_cp

[musl] [PATCH] x86_64: Generate an eh-frame entry for syscall_cp

[Ivan Trofimov]

Some eBPF-based profilers (open-telemetry/opentelemetry-ebpf-profiler, 
yandex/perforator, parca-dev/parca etc. at github.com) leverage eh-frame 
section to unwind the stacks of native executables in kernel context.

This approach works pretty well most of the time, but sometimes fails to 
unwind through hand-written assembly due to it missing cfi-* directives.
Usually it's not that big of a deal and doesn't change the whole picture 
much, however with musl things are different, since these unwinders fail 
to unwind syscalls back to userspace, as musl doesn't generate an 
eh-frame entry for __syscall_cp function.

Imagine a flamegraph, where all (or most) of the syscalls are completely 
out of place: this is the problem this patch aims to fix for x86-64.

Signed-off-by: Ivan Trofimov <i.trofimow@yandex.ru>

---
  src/thread/x86_64/syscall_cp.s | 3 +++
  1 file changed, 3 insertions(+)

diff --git a/src/thread/x86_64/syscall_cp.s b/src/thread/x86_64/syscall_cp.s
index 4f101716..296415d7 100644
--- a/src/thread/x86_64/syscall_cp.s
+++ b/src/thread/x86_64/syscall_cp.s
@@ -10,6 +10,7 @@
  .hidden __syscall_cp_asm
  .type   __syscall_cp_asm,@function
  __syscall_cp_asm:
+.cfi_startproc

  __cp_begin:
         mov (%rdi),%eax
@@ -29,3 +30,5 @@ __cp_end:
         ret
  __cp_cancel:
         jmp __cancel
+
+.cfi_endproc
--

Re: [musl] [PATCH] x86_64: Generate an eh-frame entry for syscall_cp

[Timo Teras]

On Mon, 24 Feb 2025 16:35:54 +0300
Ivan Trofimov <i.trofimow@yandex.ru> wrote:

> Some eBPF-based profilers
> (open-telemetry/opentelemetry-ebpf-profiler, yandex/perforator,
> parca-dev/parca etc. at github.com) leverage eh-frame section to
> unwind the stacks of native executables in kernel context.
>
> This approach works pretty well most of the time, but sometimes fails
> to unwind through hand-written assembly due to it missing cfi-*
> directives. Usually it's not that big of a deal and doesn't change
> the whole picture much, however with musl things are different, since
> these unwinders fail to unwind syscalls back to userspace, as musl
> doesn't generate an eh-frame entry for __syscall_cp function.

Musl has tools/add-cfi.* files to add the cfi annotations
programmatically. However, .eh_frame is intentionally turned off, and
the cfi ends up in .debug_frame only.

I earlier tried to get some .eh_frame enabled, but that led no where
and I got no responses on what could be done to enable this for
profiling purposes. Basically the response was that .eh_frame enables
throwing exception through C library which is UB and intentionally not
supported, and that it bloats the LOAD segment size for no good reason.
See also
eg. https://www.openwall.com/lists/musl/2021/07/16/1 probably there's
some other threads also.

I wonder if the newer GNU .sframe would be something that can be
enabled? Rich, any comments on enabling .sframe generation? It would
not enable throwing exceptions etc., and is smaller in size.

> Imagine a flamegraph, where all (or most) of the syscalls are
> completely out of place: this is the problem this patch aims to fix
> for x86-64.

Because of the above, opentelemetry-ebpf-profiler supports also
.debug_frame directly in the .so and in detached debug files. Try
installing musl debug data file (if available) and things will start
working a lot better.

Timo