[ruby-core:120336] [Ruby master Bug#20972] OpenSSL Memory Usage

[ruby-core:120336] [Ruby master Bug#20972] OpenSSL Memory Usage

[mrkgrandjean (Mark Grandjean) via ruby-core]

Issue #20972 has been reported by mrkgrandjean (Mark Grandjean).

----------------------------------------
Bug #20972: OpenSSL Memory Usage
https://bugs.ruby-lang.org/issues/20972

* Author: mrkgrandjean (Mark Grandjean)
* Status: Open
* ruby -v: 3.3.6
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads.
Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426

Using the same test code from that ticket 
``` ruby
require "http"
require "memory_profiler"
require "stringio"

body = StringIO.new("a" * 5*1024*1024)

MemoryProfiler.report do
  HTTP.post("https://example.com", body: body)
end.pretty_print
```
in ruby 2.6.10

```
allocated memory by gem
-----------------------------------
   1913287  unicode_normalize
   1053463  llhttp-ffi-0.5.0
    187849  http-5.2.0
     50168  openssl
     36180  addressable-2.8.7
      8904  ipaddr
       272  other
```

While testing in Ruby 3.3.6

```
allocated memory by gem
-----------------------------------
  16104976  openssl
   2393456  unicode_normalize
   1053880  llhttp-ffi-0.5.0
    151633  http-5.2.0
      9813  addressable-2.8.7
      7872  ipaddr
       240  other
        40  forwardable

```

It looks like the memory allocation issue has returned. 



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:120352] [Ruby master Bug#20972] OpenSSL Memory Usage

[luke-gru (Luke Gruber) via ruby-core]

Issue #20972 has been updated by luke-gru (Luke Gruber).


I took a look at this and this is due to 2 things that I could notice:

The call to `string.b` if it's not encoded as binary already, in the `Buffer` class in `openssl/buffering.rb`. The other one is a call to 
`rb_str_new_frozen` in `ossl_ssl_write_internal`.

Getting rid of the `rb_str_new_frozen` call brings the allocated memory down to ~ `5639000` and removing the other one gets it down to
`42840`.

I'll leave it to someone with more familiarity with the openssl codebase to provide a good fix, if it's possible, possibly using `locktmp` or something for
the frozen case. Maybe it's also possible to just force the encoding as binary for the provided string instead of duplicating it.


----------------------------------------
Bug #20972: OpenSSL Memory Usage
https://bugs.ruby-lang.org/issues/20972#change-111132

* Author: mrkgrandjean (Mark Grandjean)
* Status: Open
* ruby -v: 3.3.6
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads.
Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426

Using the same test code from that ticket 
``` ruby
require "http"
require "memory_profiler"
require "stringio"

body = StringIO.new("a" * 5*1024*1024)

MemoryProfiler.report do
  HTTP.post("https://example.com", body: body)
end.pretty_print
```
in ruby 2.6.10

```
allocated memory by gem
-----------------------------------
   1913287  unicode_normalize
   1053463  llhttp-ffi-0.5.0
    187849  http-5.2.0
     50168  openssl
     36180  addressable-2.8.7
      8904  ipaddr
       272  other
```

While testing in Ruby 3.3.6

```
allocated memory by gem
-----------------------------------
  16104976  openssl
   2393456  unicode_normalize
   1053880  llhttp-ffi-0.5.0
    151633  http-5.2.0
      9813  addressable-2.8.7
      7872  ipaddr
       240  other
        40  forwardable

```

It looks like the memory allocation issue has returned. 



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:120354] [Ruby master Bug#20972] OpenSSL Memory Usage

[byroot (Jean Boussier) via ruby-core]

Issue #20972 has been updated by byroot (Jean Boussier).


I think you are mostly correct, what exactly cause the memory growth is hard to point precisely (especially with `memory_profiler` because this codepath depends a lot on shared strings, so it's not fully clear which call exactly cause the unsharing.

But you are right that using `rb_str_locktmp` would avoid some allocations and probably some copying, and there are a few other small changes that can be made to rely less on string sharing, and reduce allocations a bit: https://github.com/ruby/openssl/pull/831

----------------------------------------
Bug #20972: OpenSSL Memory Usage
https://bugs.ruby-lang.org/issues/20972#change-111134

* Author: mrkgrandjean (Mark Grandjean)
* Status: Open
* ruby -v: 3.3.6
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
While testing large file uploads I noticed the OpenSSL Buffering is allocating a lot of memory for file uploads.
Similar to the issue in this ticket from a few years ago https://bugs.ruby-lang.org/issues/14426

Using the same test code from that ticket 
``` ruby
require "http"
require "memory_profiler"
require "stringio"

body = StringIO.new("a" * 5*1024*1024)

MemoryProfiler.report do
  HTTP.post("https://example.com", body: body)
end.pretty_print
```
in ruby 2.6.10

```
allocated memory by gem
-----------------------------------
   1913287  unicode_normalize
   1053463  llhttp-ffi-0.5.0
    187849  http-5.2.0
     50168  openssl
     36180  addressable-2.8.7
      8904  ipaddr
       272  other
```

While testing in Ruby 3.3.6

```
allocated memory by gem
-----------------------------------
  16104976  openssl
   2393456  unicode_normalize
   1053880  llhttp-ffi-0.5.0
    151633  http-5.2.0
      9813  addressable-2.8.7
      7872  ipaddr
       240  other
        40  forwardable

```

It looks like the memory allocation issue has returned. 



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/