[ruby-core:122947] [Ruby Feature#21539] Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process

[ruby-core:122947] [Ruby Feature#21539] Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process

[dalehamel (Dale Hamel) via ruby-core]

Issue #21539 has been reported by dalehamel (Dale Hamel).

----------------------------------------
Feature #21539: Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process
https://bugs.ruby-lang.org/issues/21539

* Author: dalehamel (Dale Hamel)
* Status: Open
----------------------------------------
While ruby does have a great API for getting stack traces within the ruby processes, as used by profilers like vernier and stackprof, there are some projects which aim to profile ruby from outside of the process.

Some examples include:

- rbspy (https://github.com/rbspy/rbspy/tree/main/ruby-structs/src)
- rbperf (https://github.com/javierhonduco/rbperf)
- open telemetry (https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/support/ebpf/ruby_tracer.ebpf.c, https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/ruby/ruby.go)


These first two take the approach of embedding the ruby headers within the profiler to be able to walk the stack.

otel's bpf exporter was relying on access to symbols (https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/202), which were removed some time ago https://github.com/ruby/ruby/pull/7459. As a result, it cannot profile newer rubies as it cannot unwind the stacks.

All of these solutions kind of take the approach of trying to reverse engineer the ruby process and rely on somewhat hacky approaches that touch internal things that might frequently move around between ruby versions, and all feel a bit brittle. I think the root of this is that there is no public, stable api for this sort of external profiling.

I'm not exactly sure what the solution should be, but it would be great for ruby to offer a recommended, supported, and stable way for external profilers (whether they are using perf api's or ptrace) to be able to:

- Obtain the ruby native stack starting point, and walk it while being able to resolve the native symbols
- Obtain a reference to the ruby execution context and walk the ruby interpreter stack
- Have all of this work in a stable and reasonable way regardless of if / which ruby jit is enabled

It would be awesome for some ruby maintainers / experts to weigh-in on what such an external API could look like. Perhaps a public header containing stable (or at least perhaps, versioned?) public symbols and perhaps structs as a starting point? This might necessitate that some fields be refactored out of existing structs, and could slow down work related to them (eg, members need to be added or removed).



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:123105] [Ruby Feature#21539] Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process

[ivoanjo (Ivo Anjo) via ruby-core]

Issue #21539 has been updated by ivoanjo (Ivo Anjo).


+1 this would be really cool as well for the [Datadog profiler](https://github.com/datadog/dd-trace-rb) as we currently rely on access to internal headers and thus are together with rbspy/rbperf on the group of "needs to be updated whenever Ruby internals change".

----------------------------------------
Feature #21539: Facilitate walking native and interpreter (and jit?) stacks from outside of the ruby process
https://bugs.ruby-lang.org/issues/21539#change-114422

* Author: dalehamel (Dale Hamel)
* Status: Open
----------------------------------------
While ruby does have a great API for getting stack traces within the ruby processes, as used by profilers like vernier and stackprof, there are some projects which aim to profile ruby from outside of the process.

Some examples include:

- rbspy (https://github.com/rbspy/rbspy/tree/main/ruby-structs/src)
- rbperf (https://github.com/javierhonduco/rbperf)
- open telemetry (https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/support/ebpf/ruby_tracer.ebpf.c, https://github.com/open-telemetry/opentelemetry-ebpf-profiler/blob/main/interpreter/ruby/ruby.go)


These first two take the approach of embedding the ruby headers within the profiler to be able to walk the stack.

otel's bpf exporter was relying on access to symbols (https://github.com/open-telemetry/opentelemetry-ebpf-profiler/issues/202), which were removed some time ago https://github.com/ruby/ruby/pull/7459. As a result, it cannot profile newer rubies as it cannot unwind the stacks.

All of these solutions kind of take the approach of trying to reverse engineer the ruby process and rely on somewhat hacky approaches that touch internal things that might frequently move around between ruby versions, and all feel a bit brittle. I think the root of this is that there is no public, stable api for this sort of external profiling.

I'm not exactly sure what the solution should be, but it would be great for ruby to offer a recommended, supported, and stable way for external profilers (whether they are using perf api's or ptrace) to be able to:

- Obtain the ruby native stack starting point, and walk it while being able to resolve the native symbols
- Obtain a reference to the ruby execution context and walk the ruby interpreter stack
- Have all of this work in a stable and reasonable way regardless of if / which ruby jit is enabled

It would be awesome for some ruby maintainers / experts to weigh-in on what such an external API could look like. Perhaps a public header containing stable (or at least perhaps, versioned?) public symbols and perhaps structs as a starting point? This might necessitate that some fields be refactored out of existing structs, and could slow down work related to them (eg, members need to be added or removed).



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/