[ruby-core:124419] [Ruby Bug#21827] Deprecating Ripper

[ruby-core:124419] [Ruby Bug#21827] Deprecating Ripper

[Eregon (Benoit Daloze) via ruby-core]

Issue #21827 has been reported by Eregon (Benoit Daloze).

----------------------------------------
Bug #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827

* Author: Eregon (Benoit Daloze)
* Status: Open
* Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN, 4.0: UNKNOWN
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124440] [Ruby Feature#21827] Deprecating Ripper

[st0012 (Stan Lo) via ruby-core]

Issue #21827 has been updated by st0012 (Stan Lo).


Ripper is at least used in these default/bundled gems:

- `irb`
- `rdoc`
- `rbs`
- `power_assert`
- `syntax_suggect`

While I agree that we should deprecate it, I also would like to avoid showing the deprecation message when users use these gems in the next Ruby release.
Can you contact related maintainers to understand how hard it'd be for the project to migrate off Ripper?

Wrt `irb` and `rdoc`, which I maintain:
- @tompng and I already decided to migrate to Prism-based parser/tokenizer for RDoc this year. So we should be good there
- `irb` uses Ripper quite extensively and doesn't just call `Ripper.lex`. So it'll probably take some work to migrate

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-115989

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124441] [Ruby Feature#21827] Deprecating Ripper

[kddnewton (Kevin Newton) via ruby-core]

Issue #21827 has been updated by kddnewton (Kevin Newton).


I don't think we're going to be able to deprecate in 4.1, Ripper is still very much used in the ecosystem. I think we can do the work over the next couple of years to remove it, but 4.1 is quite a tight timeline.

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-115990

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124532] [Ruby Feature#21827] Deprecating Ripper

[Earlopain (Earlopain _) via ruby-core]

Issue #21827 has been updated by Earlopain (Earlopain _).


Prism contains a compatiblity layer with ripper, which I think would be a good temporary solution. It's not quite there yet (irb, rdoc use undocumented APIs which it doesn't mirror yet), and there are some subtle compatibility issues (irb has 20 test failures with the translator used) but overall it's pretty good I'd say.

To demonstrate, here is a change for `power_assert`: https://github.com/ruby/power_assert/pull/65. It passes the full test suite and the effort needed is not nearly the same as porting it to prism directly. Eventually nobody should use the translator anymore but to migrate it makes things quite a bit simpler.

Currently I'm working on making the translator more compatible with the gems listed above. What do you think?

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-116100

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124551] [Ruby Feature#21827] Deprecating Ripper

[Eregon (Benoit Daloze) via ruby-core]

Issue #21827 has been updated by Eregon (Benoit Daloze).


Earlopain (Earlopain _) wrote in #note-4:
> Currently I'm working on making the translator more compatible with the gems listed above. What do you think?

Thank you, I think this is a great temporary solution and I will try to implement Ripper on TruffleRuby using that since the Ripper extension has proven to be quite problematic on TruffleRuby.

I heard during the dev meeting that it's clearly too soon to deprecate Ripper e.g. with a deprecation warning, we need more time to migrate usages.
However I'll make a PR to suggest using Prism for new code and to suggest migrating to Prism for existing code as it will likely be deprecated at some point.

I think it's valuable to e.g. migrate `irb` to Prism because that way we don't have the extra indirections of the translation layer, and the better API might allow more correctness too for what IRB does with Ripper.
I suspect the overhead is significant, and BTW original Ripper is also significantly slower than Prism as mentioned in the description.
This is the issue to migrate to Prism for IRB: https://github.com/ruby/irb/issues/1024
I and others might be able to help a bit with that, let me know how (cc @st0012). I guess first we should merge the existing PRs from @tompng.

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-116122

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124555] [Ruby Feature#21827] Deprecating Ripper

[Earlopain (Earlopain _) via ruby-core]

Issue #21827 has been updated by Earlopain (Earlopain _).


> the better API might allow more correctness too for what IRB does with Ripper

I would think that irb has long since dealt with the particularities of ripper. It's already implemented now, I don't think correctness would improve noticably by using something else.

> I suspect the overhead is significant

The ripper translator is about 3x slower (tested with `Ripper.sexp` over prism ruby code). Maybe there are some optimizations to improve that a bit, but haven't checked yet. The translation for the `parser` gem is able to be faster than the original gem, so I'm hopeful.

> However I'll make a PR to suggest using Prism for new code

👍

> I think it's valuable to e.g. migrate irb to Prism

Yes, it is. Using the translation should not be seen as the last step. But it can be a huge benefit to not be left behind/hold up the deprecation for too long. RuboCop for example where I finished the migration to the parser translator only took some time to improve the translator in prism and then it was just switching out some classes that rubocop uses. Otherwise it would have taken years (maybe/maybe not exagerating) to migrate all the code. Now it can be done without any real pressure (the `parser` gem does not adapt to new syntax anymore)

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-116127

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124583] [Ruby Feature#21827] Deprecating Ripper

[tompng (tomoya ishida) via ruby-core]

Issue #21827 has been updated by tompng (tomoya ishida).


For IRB, I think migration pull-request is almost done and it won't take so long to completely migrate to Prism.

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-116162

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper, which feels as a result like a major hack.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124628] [Ruby Feature#21827] Deprecating Ripper

[Earlopain (Earlopain _) via ruby-core]

Issue #21827 has been updated by Earlopain (Earlopain _).


I created https://github.com/ruby/rbs/pull/2828 to stop using `ripper` in `rbs`. It used ripper only in two places to extract comments so it wasn't difficult to migrate.

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-116214

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:124904] [Ruby Feature#21827] Deprecating Ripper

[Earlopain (Earlopain _) via ruby-core]

Issue #21827 has been updated by Earlopain (Earlopain _).


`rdoc` and `irb` have now stopped using `ripper`. https://github.com/ruby/syntax_suggest/pull/251 is for `syntax_suggest`.

That leaves just `power_assert` for the ruby gems. I gave that a quick look but it's difficult to understand for me what the current ripper code is actually doing. I currently still have the PR for using the translator open (which I find acceptable to do) but I wouldn't mind someone else giving it a shot to use native prism.

BTW, truffleruby has replaced its ripper implementation with the prism translator in https://github.com/truffleruby/truffleruby/pull/4102.

----------------------------------------
Feature #21827: Deprecating Ripper
https://bugs.ruby-lang.org/issues/21827#change-116567

* Author: Eregon (Benoit Daloze)
* Status: Open
----------------------------------------
I think it is time to consider deprecating `Ripper`.
[Matz has agreed that going forward the official parser API for Ruby will be the Prism API](https://railsatscale.com/2024-04-16-prism-in-2024/) so it's clear the official Ruby parsing API is the Prism API.

`Ripper` while being currently supported on CRuby, JRuby and TruffleRuby (partially) has a significant maintenance overhead.
For CRuby, it roughly doubles the number of lines in `parse.y` and adds lots of `#ifdef RIPPER`/`#ifndef RIPPER` which makes it really hard to follow.
For JRuby it's many thousands of lines of code to maintain just for that: https://github.com/jruby/jruby/tree/master/core/src/main/java/org/jruby/ext/ripper
For TruffleRuby, `ripper` is by far the most difficult C extension to support as it relies on many internal functions. When an internal function is used by ripper is missing it kills the entire process. On every update of `RUBY_VERSION` in TruffleRuby, the biggest amount of time is spent dealing with Ripper.

It could be possible to implement Ripper based on Prism, in fact there is [some support for that](https://github.com/ruby/prism/blob/main/docs/ripper_translation.md), but the fact the Ripper API is rather unstable means it's hard to be compatible, and currently the translation layer [relies on original ripper for some things](https://github.com/ruby/prism/blob/e0e5e68acbed98a6b4e4e77b9b52b4488719e2ca/lib/prism/translation/ripper.rb#L4).

It's also worth noting that [`Ripper.sexp` is 2.75x slower than `Prism.parse`](https://eregon.me/blog/2024/10/27/benchmarking-ruby-parsers.html#parsing-and-not-walking), while providing a much worse API (widely recognized as such, e.g. `Ripper.sexp("def m = 42")` vs `Prism.parse("def m = 42")`, lots of `nil` with unclear meaning in `Ripper`).
So from a new code perspective there seems to be very little value to use `Ripper`.

`Ripper` has lexing via `Ripper.lex`, which can be replaced by `Prism::lex_compat`.

I think we should start to deprecate `Ripper`.

The advantages of deprecating it are:
* Make it clear that this API should not be used in new code
* Encourage the usages to migrate to the Prism API, which is stable, officially supported, designed with many people to be usable and convenient for every parsing/tooling usage, etc.
* Improve portability of Ruby code between Ruby implementations (given Ripper is not fully compatible between Ruby implementations)

I propose to deprecate Ripper in 4.1.
I'm not sure when it would be appropriate to remove it given it's a more established API, maybe in 4.2, 4.3?



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/