* [ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced
@ 2024-07-30 16:29 alanwu (Alan Wu) via ruby-core
2024-07-31 2:37 ` [ruby-core:118754] " mame (Yusuke Endoh) via ruby-core
` (3 more replies)
0 siblings, 4 replies; 5+ messages in thread
From: alanwu (Alan Wu) via ruby-core @ 2024-07-30 16:29 UTC (permalink / raw)
To: ruby-core; +Cc: alanwu (Alan Wu)
Issue #20659 has been reported by alanwu (Alan Wu).
----------------------------------------
Bug #20659: Speed regression of `parse.y` parser after numeric nodes were introduced
https://bugs.ruby-lang.org/issues/20659
* Author: alanwu (Alan Wu)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
The [`mail` benchmark from yjit-bench][1] is about 20% slower on `master` compared to commit:98eeadc9 ("Development of 3.4.0 started.") as the baseline, comparing running time of the Ruby process running the benchmark for a single iteration. Much of this workload is Ruby parsing.
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby' '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.0 ms ± 2.0 ms [User: 776.9 ms, System: 111.7 ms]
Range (min … max): 885.7 ms … 892.1 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 891.7 ms ± 1.6 ms [User: 794.5 ms, System: 96.9 ms]
Range (min … max): 889.0 ms … 894.7 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.086 s ± 0.003 s [User: 0.951 s, System: 0.134 s]
Range (min … max): 1.080 s … 1.091 s 10 runs
Summary
'~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ran
1.00 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.22 ± 0.00 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
$ for tag in 3.3-equiv master before-numeric-nodes; do echo $tag: $(~/.rubies/$tag/bin/ruby -v); done
3.3-equiv: ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [x86_64-linux]
master: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
before-numeric-nodes: ruby 3.4.0dev (2024-01-06T18:26:38Z master 76afbda5b5) [x86_64-linux]
```
Using Valgrind's DHAT reveals that commit:1b8d01136c3ff6c60325c7609d61e19ac42acd9f ("Introduce Numeric Node's") issues roughly 3 times more `malloc` calls compared to the baseline, most of them comping from strdup() calls in set_number_literal().
Comparison with `--parser=prism`, for interest:
<details>
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby,master/bin/ruby --parser=prism' \
'~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.7 ms ± 2.6 ms [User: 771.4 ms, System: 118.0 ms]
Range (min … max): 885.8 ms … 894.6 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 890.4 ms ± 1.6 ms [User: 776.8 ms, System: 113.3 ms]
Range (min … max): 888.4 ms … 892.9 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.087 s ± 0.004 s [User: 0.968 s, System: 0.119 s]
Range (min … max): 1.083 s … 1.097 s 10 runs
Benchmark 4: ~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 826.9 ms ± 2.1 ms [User: 725.8 ms, System: 100.7 ms]
Range (min … max): 823.6 ms … 830.9 ms 10 runs
Summary
'~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb' ran
1.08 ± 0.00 times faster than '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.08 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.31 ± 0.01 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
```
</details>
[1]: https://github.com/Shopify/yjit-bench/blob/b8f1bd8931d4757afdae933ef6f11e0767ba90b8/benchmarks/mail/benchmark.rb
--
https://bugs.ruby-lang.org/
______________________________________________
ruby-core mailing list -- ruby-core@ml.ruby-lang.org
To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:118754] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced
2024-07-30 16:29 [ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced alanwu (Alan Wu) via ruby-core
@ 2024-07-31 2:37 ` mame (Yusuke Endoh) via ruby-core
2024-07-31 3:34 ` [ruby-core:118756] " mame (Yusuke Endoh) via ruby-core
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: mame (Yusuke Endoh) via ruby-core @ 2024-07-31 2:37 UTC (permalink / raw)
To: ruby-core; +Cc: mame (Yusuke Endoh)
Issue #20659 has been updated by mame (Yusuke Endoh).
Status changed from Open to Assigned
Assignee set to yui-knk (Kaneko Yuichiro)
----------------------------------------
Bug #20659: Speed regression of `parse.y` parser after numeric nodes were introduced
https://bugs.ruby-lang.org/issues/20659#change-109300
* Author: alanwu (Alan Wu)
* Status: Assigned
* Assignee: yui-knk (Kaneko Yuichiro)
* ruby -v: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
The [`mail` benchmark from yjit-bench][1] is about 20% slower on `master` compared to commit:98eeadc9 ("Development of 3.4.0 started.") as the baseline, comparing running time of the Ruby process running the benchmark for a single iteration. Much of this workload is Ruby parsing.
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby' '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.0 ms ± 2.0 ms [User: 776.9 ms, System: 111.7 ms]
Range (min … max): 885.7 ms … 892.1 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 891.7 ms ± 1.6 ms [User: 794.5 ms, System: 96.9 ms]
Range (min … max): 889.0 ms … 894.7 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.086 s ± 0.003 s [User: 0.951 s, System: 0.134 s]
Range (min … max): 1.080 s … 1.091 s 10 runs
Summary
'~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ran
1.00 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.22 ± 0.00 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
$ for tag in 3.3-equiv master before-numeric-nodes; do echo $tag: $(~/.rubies/$tag/bin/ruby -v); done
3.3-equiv: ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [x86_64-linux]
master: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
before-numeric-nodes: ruby 3.4.0dev (2024-01-06T18:26:38Z master 76afbda5b5) [x86_64-linux]
```
Using Valgrind's DHAT reveals that commit:1b8d01136c3ff6c60325c7609d61e19ac42acd9f ("Introduce Numeric Node's") issues roughly 3 times more malloc(3) calls compared to the baseline, most of them coming from strdup() calls in set_number_literal().
Comparison with `--parser=prism`, for interest:
<details>
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby,master/bin/ruby --parser=prism' \
'~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.7 ms ± 2.6 ms [User: 771.4 ms, System: 118.0 ms]
Range (min … max): 885.8 ms … 894.6 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 890.4 ms ± 1.6 ms [User: 776.8 ms, System: 113.3 ms]
Range (min … max): 888.4 ms … 892.9 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.087 s ± 0.004 s [User: 0.968 s, System: 0.119 s]
Range (min … max): 1.083 s … 1.097 s 10 runs
Benchmark 4: ~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 826.9 ms ± 2.1 ms [User: 725.8 ms, System: 100.7 ms]
Range (min … max): 823.6 ms … 830.9 ms 10 runs
Summary
'~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb' ran
1.08 ± 0.00 times faster than '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.08 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.31 ± 0.01 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
```
</details>
[1]: https://github.com/Shopify/yjit-bench/blob/b8f1bd8931d4757afdae933ef6f11e0767ba90b8/benchmarks/mail/benchmark.rb
--
https://bugs.ruby-lang.org/
______________________________________________
ruby-core mailing list -- ruby-core@ml.ruby-lang.org
To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:118756] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced
2024-07-30 16:29 [ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced alanwu (Alan Wu) via ruby-core
2024-07-31 2:37 ` [ruby-core:118754] " mame (Yusuke Endoh) via ruby-core
@ 2024-07-31 3:34 ` mame (Yusuke Endoh) via ruby-core
2024-07-31 12:38 ` [ruby-core:118759] " alanwu (Alan Wu) via ruby-core
2024-12-12 17:29 ` [ruby-core:120221] " alanwu (Alan Wu) via ruby-core
3 siblings, 0 replies; 5+ messages in thread
From: mame (Yusuke Endoh) via ruby-core @ 2024-07-31 3:34 UTC (permalink / raw)
To: ruby-core; +Cc: mame (Yusuke Endoh)
Issue #20659 has been updated by mame (Yusuke Endoh).
Just a simple curiosity, I am not sure how the change will affect the performance so much. Does mail gem use `eval` a lot?
----------------------------------------
Bug #20659: Speed regression of `parse.y` parser after numeric nodes were introduced
https://bugs.ruby-lang.org/issues/20659#change-109302
* Author: alanwu (Alan Wu)
* Status: Assigned
* Assignee: yui-knk (Kaneko Yuichiro)
* ruby -v: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
The [`mail` benchmark from yjit-bench][1] is about 20% slower on `master` compared to commit:98eeadc9 ("Development of 3.4.0 started.") as the baseline, comparing running time of the Ruby process running the benchmark for a single iteration. Much of this workload is Ruby parsing.
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby' '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.0 ms ± 2.0 ms [User: 776.9 ms, System: 111.7 ms]
Range (min … max): 885.7 ms … 892.1 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 891.7 ms ± 1.6 ms [User: 794.5 ms, System: 96.9 ms]
Range (min … max): 889.0 ms … 894.7 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.086 s ± 0.003 s [User: 0.951 s, System: 0.134 s]
Range (min … max): 1.080 s … 1.091 s 10 runs
Summary
'~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ran
1.00 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.22 ± 0.00 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
$ for tag in 3.3-equiv master before-numeric-nodes; do echo $tag: $(~/.rubies/$tag/bin/ruby -v); done
3.3-equiv: ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [x86_64-linux]
master: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
before-numeric-nodes: ruby 3.4.0dev (2024-01-06T18:26:38Z master 76afbda5b5) [x86_64-linux]
```
Using Valgrind's DHAT reveals that commit:1b8d01136c3ff6c60325c7609d61e19ac42acd9f ("Introduce Numeric Node's") issues roughly 3 times more malloc(3) calls compared to the baseline, most of them coming from strdup() calls in set_number_literal().
Comparison with `--parser=prism`, for interest:
<details>
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby,master/bin/ruby --parser=prism' \
'~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.7 ms ± 2.6 ms [User: 771.4 ms, System: 118.0 ms]
Range (min … max): 885.8 ms … 894.6 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 890.4 ms ± 1.6 ms [User: 776.8 ms, System: 113.3 ms]
Range (min … max): 888.4 ms … 892.9 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.087 s ± 0.004 s [User: 0.968 s, System: 0.119 s]
Range (min … max): 1.083 s … 1.097 s 10 runs
Benchmark 4: ~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 826.9 ms ± 2.1 ms [User: 725.8 ms, System: 100.7 ms]
Range (min … max): 823.6 ms … 830.9 ms 10 runs
Summary
'~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb' ran
1.08 ± 0.00 times faster than '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.08 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.31 ± 0.01 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
```
</details>
[1]: https://github.com/Shopify/yjit-bench/blob/b8f1bd8931d4757afdae933ef6f11e0767ba90b8/benchmarks/mail/benchmark.rb
--
https://bugs.ruby-lang.org/
______________________________________________
ruby-core mailing list -- ruby-core@ml.ruby-lang.org
To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:118759] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced
2024-07-30 16:29 [ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced alanwu (Alan Wu) via ruby-core
2024-07-31 2:37 ` [ruby-core:118754] " mame (Yusuke Endoh) via ruby-core
2024-07-31 3:34 ` [ruby-core:118756] " mame (Yusuke Endoh) via ruby-core
@ 2024-07-31 12:38 ` alanwu (Alan Wu) via ruby-core
2024-12-12 17:29 ` [ruby-core:120221] " alanwu (Alan Wu) via ruby-core
3 siblings, 0 replies; 5+ messages in thread
From: alanwu (Alan Wu) via ruby-core @ 2024-07-31 12:38 UTC (permalink / raw)
To: ruby-core; +Cc: alanwu (Alan Wu)
Issue #20659 has been updated by alanwu (Alan Wu).
Some light profiling shows that this workload spends over half of its time in `Kernel#require`. I don't think it's coming from `eval`s in the body of the benchmark, but rather the loading of it. Running one iteration is important to trigger all the autoloads.
The gem does contain a large number of integer literals in the generated parsers it bundles. For example: <https://raw.githubusercontent.com/mikel/mail/master/lib/mail/parsers/address_lists_parser.rb>
----------------------------------------
Bug #20659: Speed regression of `parse.y` parser after numeric nodes were introduced
https://bugs.ruby-lang.org/issues/20659#change-109305
* Author: alanwu (Alan Wu)
* Status: Assigned
* Assignee: yui-knk (Kaneko Yuichiro)
* ruby -v: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
The [`mail` benchmark from yjit-bench][1] is about 20% slower on `master` compared to commit:98eeadc9 ("Development of 3.4.0 started.") as the baseline, comparing running time of the Ruby process running the benchmark for a single iteration. Much of this workload is Ruby parsing.
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby' '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.0 ms ± 2.0 ms [User: 776.9 ms, System: 111.7 ms]
Range (min … max): 885.7 ms … 892.1 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 891.7 ms ± 1.6 ms [User: 794.5 ms, System: 96.9 ms]
Range (min … max): 889.0 ms … 894.7 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.086 s ± 0.003 s [User: 0.951 s, System: 0.134 s]
Range (min … max): 1.080 s … 1.091 s 10 runs
Summary
'~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ran
1.00 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.22 ± 0.00 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
$ for tag in 3.3-equiv master before-numeric-nodes; do echo $tag: $(~/.rubies/$tag/bin/ruby -v); done
3.3-equiv: ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [x86_64-linux]
master: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
before-numeric-nodes: ruby 3.4.0dev (2024-01-06T18:26:38Z master 76afbda5b5) [x86_64-linux]
```
Using Valgrind's DHAT reveals that commit:1b8d01136c3ff6c60325c7609d61e19ac42acd9f ("Introduce Numeric Node's") issues roughly 3 times more malloc(3) calls compared to the baseline, most of them coming from strdup() calls in set_number_literal().
Comparison with `--parser=prism`, for interest:
<details>
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby,master/bin/ruby --parser=prism' \
'~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.7 ms ± 2.6 ms [User: 771.4 ms, System: 118.0 ms]
Range (min … max): 885.8 ms … 894.6 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 890.4 ms ± 1.6 ms [User: 776.8 ms, System: 113.3 ms]
Range (min … max): 888.4 ms … 892.9 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.087 s ± 0.004 s [User: 0.968 s, System: 0.119 s]
Range (min … max): 1.083 s … 1.097 s 10 runs
Benchmark 4: ~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 826.9 ms ± 2.1 ms [User: 725.8 ms, System: 100.7 ms]
Range (min … max): 823.6 ms … 830.9 ms 10 runs
Summary
'~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb' ran
1.08 ± 0.00 times faster than '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.08 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.31 ± 0.01 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
```
</details>
[1]: https://github.com/Shopify/yjit-bench/blob/b8f1bd8931d4757afdae933ef6f11e0767ba90b8/benchmarks/mail/benchmark.rb
--
https://bugs.ruby-lang.org/
______________________________________________
ruby-core mailing list -- ruby-core@ml.ruby-lang.org
To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
* [ruby-core:120221] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced
2024-07-30 16:29 [ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced alanwu (Alan Wu) via ruby-core
` (2 preceding siblings ...)
2024-07-31 12:38 ` [ruby-core:118759] " alanwu (Alan Wu) via ruby-core
@ 2024-12-12 17:29 ` alanwu (Alan Wu) via ruby-core
3 siblings, 0 replies; 5+ messages in thread
From: alanwu (Alan Wu) via ruby-core @ 2024-12-12 17:29 UTC (permalink / raw)
To: ruby-core; +Cc: alanwu (Alan Wu)
Issue #20659 has been updated by alanwu (Alan Wu).
Status changed from Assigned to Closed
I believe this was fixed by commit:c93d07ed7448f332379cf21b4b7b649b057e5671.
----------------------------------------
Bug #20659: Speed regression of `parse.y` parser after numeric nodes were introduced
https://bugs.ruby-lang.org/issues/20659#change-110985
* Author: alanwu (Alan Wu)
* Status: Closed
* Assignee: yui-knk (Kaneko Yuichiro)
* ruby -v: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
The [`mail` benchmark from yjit-bench][1] is about 20% slower on `master` compared to commit:98eeadc9 ("Development of 3.4.0 started.") as the baseline, comparing running time of the Ruby process running the benchmark for a single iteration. Much of this workload is Ruby parsing.
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby' '~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.0 ms ± 2.0 ms [User: 776.9 ms, System: 111.7 ms]
Range (min … max): 885.7 ms … 892.1 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 891.7 ms ± 1.6 ms [User: 794.5 ms, System: 96.9 ms]
Range (min … max): 889.0 ms … 894.7 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.086 s ± 0.003 s [User: 0.951 s, System: 0.134 s]
Range (min … max): 1.080 s … 1.091 s 10 runs
Summary
'~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb' ran
1.00 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.22 ± 0.00 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
$ for tag in 3.3-equiv master before-numeric-nodes; do echo $tag: $(~/.rubies/$tag/bin/ruby -v); done
3.3-equiv: ruby 3.4.0dev (2023-12-25T09:13:40Z master 98eeadc932) [x86_64-linux]
master: ruby 3.4.0dev (2024-07-30T14:01:43Z master 1164b6a7ba) [x86_64-linux]
before-numeric-nodes: ruby 3.4.0dev (2024-01-06T18:26:38Z master 76afbda5b5) [x86_64-linux]
```
Using Valgrind's DHAT reveals that commit:1b8d01136c3ff6c60325c7609d61e19ac42acd9f ("Introduce Numeric Node's") issues roughly 3 times more malloc(3) calls compared to the baseline, most of them coming from strdup() calls in set_number_literal().
Comparison with `--parser=prism`, for interest:
<details>
```
$ hyperfine -L ruby '3.3-equiv/bin/ruby,before-numeric-nodes/bin/ruby,master/bin/ruby,master/bin/ruby --parser=prism' \
'~/.rubies/{ruby} -Iharness-once benchmarks/mail/benchmark.rb'
Benchmark 1: ~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 889.7 ms ± 2.6 ms [User: 771.4 ms, System: 118.0 ms]
Range (min … max): 885.8 ms … 894.6 ms 10 runs
Benchmark 2: ~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 890.4 ms ± 1.6 ms [User: 776.8 ms, System: 113.3 ms]
Range (min … max): 888.4 ms … 892.9 ms 10 runs
Benchmark 3: ~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 1.087 s ± 0.004 s [User: 0.968 s, System: 0.119 s]
Range (min … max): 1.083 s … 1.097 s 10 runs
Benchmark 4: ~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb
Time (mean ± σ): 826.9 ms ± 2.1 ms [User: 725.8 ms, System: 100.7 ms]
Range (min … max): 823.6 ms … 830.9 ms 10 runs
Summary
'~/.rubies/master/bin/ruby --parser=prism -Iharness-once benchmarks/mail/benchmark.rb' ran
1.08 ± 0.00 times faster than '~/.rubies/3.3-equiv/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.08 ± 0.00 times faster than '~/.rubies/before-numeric-nodes/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
1.31 ± 0.01 times faster than '~/.rubies/master/bin/ruby -Iharness-once benchmarks/mail/benchmark.rb'
```
</details>
[1]: https://github.com/Shopify/yjit-bench/blob/b8f1bd8931d4757afdae933ef6f11e0767ba90b8/benchmarks/mail/benchmark.rb
--
https://bugs.ruby-lang.org/
______________________________________________
ruby-core mailing list -- ruby-core@ml.ruby-lang.org
To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-12-12 17:29 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-07-30 16:29 [ruby-core:118750] [Ruby master Bug#20659] Speed regression of `parse.y` parser after numeric nodes were introduced alanwu (Alan Wu) via ruby-core
2024-07-31 2:37 ` [ruby-core:118754] " mame (Yusuke Endoh) via ruby-core
2024-07-31 3:34 ` [ruby-core:118756] " mame (Yusuke Endoh) via ruby-core
2024-07-31 12:38 ` [ruby-core:118759] " alanwu (Alan Wu) via ruby-core
2024-12-12 17:29 ` [ruby-core:120221] " alanwu (Alan Wu) via ruby-core
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).