[ruby-core:122382] [Ruby Bug#21394] Memory leak in Prism's RubyVM::InstructionSequence.new

[ruby-core:122382] [Ruby Bug#21394] Memory leak in Prism's RubyVM::InstructionSequence.new

[peterzhu2118 (Peter Zhu) via ruby-core]

Issue #21394 has been reported by peterzhu2118 (Peter Zhu).

----------------------------------------
Bug #21394: Memory leak in Prism's RubyVM::InstructionSequence.new
https://bugs.ruby-lang.org/issues/21394

* Author: peterzhu2118 (Peter Zhu)
* Status: Open
* Backport: 3.2: DONTNEED, 3.3: DONTNEED, 3.4: REQUIRED
----------------------------------------
Fix: https://github.com/ruby/ruby/pull/13496

There are two ways to make RubyVM::InstructionSequence.new raise which would cause the options->scopes to leak memory:

1. Passing in any (non T_FILE) object where the to_str raises.
2. Passing in a T_FILE object where String#initialize_dup raises. This is because rb_io_path dups the string.

Example 1:

```ruby
10.times do
  100_000.times do
    RubyVM::InstructionSequence.new(nil)
  rescue TypeError
  end

  puts `ps -o rss= -p #{$$}`
end
```

Before:

```
13392
17104
20256
23920
27264
30432
33584
36752
40032
43232
```

After:

```
9392
11072
11648
11648
11648
11712
11712
11712
11744
11744
```

Example 2:

```ruby
require "tempfile"

MyError = Class.new(StandardError)
String.prepend(Module.new do
  def initialize_dup(_)
    if $raise_on_dup
      raise MyError
    else
      super
    end
  end
end)

Tempfile.create do |f|
  10.times do
    100_000.times do
      $raise_on_dup = true
      RubyVM::InstructionSequence.new(f)
    rescue MyError
    else
      raise "MyError was not raised during RubyVM::InstructionSequence.new"
    end

    puts `ps -o rss= -p #{$$}`
  ensure
    $raise_on_dup = false
  end
end
```

Before:

```
14080
18512
22000
25184
28320
31600
34736
37904
41088
44256
```

After:

```
12016
12464
12880
12880
12880
12912
12912
12912
12912
12912
```



-- 
https://bugs.ruby-lang.org/
______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/