ruby-core@ruby-lang.org archive (unofficial mirror)
 help / color / mirror / Atom feed
* [ruby-core:120058] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
@ 2024-11-29 16:07 javanthropus (Jeremy Bopp) via ruby-core
  2024-11-29 16:35 ` [ruby-core:120059] " javanthropus (Jeremy Bopp) via ruby-core
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: javanthropus (Jeremy Bopp) via ruby-core @ 2024-11-29 16:07 UTC (permalink / raw)
  To: ruby-core; +Cc: javanthropus (Jeremy Bopp)

Issue #20924 has been reported by javanthropus (Jeremy Bopp).

----------------------------------------
Bug #20924: IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
https://bugs.ruby-lang.org/issues/20924

* Author: javanthropus (Jeremy Bopp)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-11-28T12:38:16Z master 3af1a04741) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
```ruby
require 'tempfile'

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end
```

This doesn't happen with UTF-32BE.  This also doesn't happen in Ruby 3.3, but it does happen in 3.4-dev and Ruby 3.0 - 3.2.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ruby-core:120059] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
  2024-11-29 16:07 [ruby-core:120058] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character javanthropus (Jeremy Bopp) via ruby-core
@ 2024-11-29 16:35 ` javanthropus (Jeremy Bopp) via ruby-core
  2024-12-15  9:33 ` [ruby-core:120248] " nagachika (Tomoyuki Chikanaga) via ruby-core
  2025-01-15  1:53 ` [ruby-core:120686] " k0kubun (Takashi Kokubun) via ruby-core
  2 siblings, 0 replies; 4+ messages in thread
From: javanthropus (Jeremy Bopp) via ruby-core @ 2024-11-29 16:35 UTC (permalink / raw)
  To: ruby-core; +Cc: javanthropus (Jeremy Bopp)

Issue #20924 has been updated by javanthropus (Jeremy Bopp).


Note that replacing `#readline` with `#gets` in the examples causes the issue to be seen in all Ruby versions, including 3.3.6.

----------------------------------------
Bug #20924: IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
https://bugs.ruby-lang.org/issues/20924#change-110797

* Author: javanthropus (Jeremy Bopp)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-11-28T12:38:16Z master 3af1a04741) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
```ruby
require 'tempfile'

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end
```

This doesn't happen with UTF-32BE.  This also doesn't happen in Ruby 3.3, but it does happen in 3.4-dev and Ruby 3.0 - 3.2.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ruby-core:120248] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
  2024-11-29 16:07 [ruby-core:120058] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character javanthropus (Jeremy Bopp) via ruby-core
  2024-11-29 16:35 ` [ruby-core:120059] " javanthropus (Jeremy Bopp) via ruby-core
@ 2024-12-15  9:33 ` nagachika (Tomoyuki Chikanaga) via ruby-core
  2025-01-15  1:53 ` [ruby-core:120686] " k0kubun (Takashi Kokubun) via ruby-core
  2 siblings, 0 replies; 4+ messages in thread
From: nagachika (Tomoyuki Chikanaga) via ruby-core @ 2024-12-15  9:33 UTC (permalink / raw)
  To: ruby-core; +Cc: nagachika (Tomoyuki Chikanaga)

Issue #20924 has been updated by nagachika (Tomoyuki Chikanaga).

Backport changed from 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED to 3.1: REQUIRED, 3.2: DONE, 3.3: REQUIRED

ruby_3_2 commit:3110d5c8abf8f710de42989fbc35870287a12f75 merged revision(s) commit:e90b447655dd39ad1eb645cdaae450efd605db00.

----------------------------------------
Bug #20924: IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
https://bugs.ruby-lang.org/issues/20924#change-111018

* Author: javanthropus (Jeremy Bopp)
* Status: Closed
* ruby -v: ruby 3.4.0dev (2024-11-28T12:38:16Z master 3af1a04741) +PRISM [x86_64-linux]
* Backport: 3.1: REQUIRED, 3.2: DONE, 3.3: REQUIRED
----------------------------------------
```ruby
require 'tempfile'

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end
```

This doesn't happen with UTF-32BE.  This also doesn't happen in Ruby 3.3, but it does happen in 3.4-dev and Ruby 3.0 - 3.2.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

* [ruby-core:120686] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
  2024-11-29 16:07 [ruby-core:120058] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character javanthropus (Jeremy Bopp) via ruby-core
  2024-11-29 16:35 ` [ruby-core:120059] " javanthropus (Jeremy Bopp) via ruby-core
  2024-12-15  9:33 ` [ruby-core:120248] " nagachika (Tomoyuki Chikanaga) via ruby-core
@ 2025-01-15  1:53 ` k0kubun (Takashi Kokubun) via ruby-core
  2 siblings, 0 replies; 4+ messages in thread
From: k0kubun (Takashi Kokubun) via ruby-core @ 2025-01-15  1:53 UTC (permalink / raw)
  To: ruby-core; +Cc: k0kubun (Takashi Kokubun)

Issue #20924 has been updated by k0kubun (Takashi Kokubun).

Backport changed from 3.1: REQUIRED, 3.2: DONE, 3.3: REQUIRED to 3.1: REQUIRED, 3.2: DONE, 3.3: DONE

ruby_3_3 commit:00147cbab567b05b8a4137875bbda341ef704760 merged revision(s) commit:e90b447655dd39ad1eb645cdaae450efd605db00.

----------------------------------------
Bug #20924: IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character
https://bugs.ruby-lang.org/issues/20924#change-111509

* Author: javanthropus (Jeremy Bopp)
* Status: Closed
* ruby -v: ruby 3.4.0dev (2024-11-28T12:38:16Z master 3af1a04741) +PRISM [x86_64-linux]
* Backport: 3.1: REQUIRED, 3.2: DONE, 3.3: DONE
----------------------------------------
```ruby
require 'tempfile'

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(binmode: true, encoding: 'utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit that would truncate a character becomes completely ignored
  f.readline(3).bytesize  # => 40; should be 4
end

Tempfile.open(encoding: 'utf-8:utf-32le') do |f|
  f.write('0123456789')
  f.rewind

  # A limit on character boundaries is respected
  f.readline(4).bytesize  # => 4
end
```

This doesn't happen with UTF-32BE.  This also doesn't happen in Ruby 3.3, but it does happen in 3.4-dev and Ruby 3.0 - 3.2.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2025-01-15  1:54 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-11-29 16:07 [ruby-core:120058] [Ruby master Bug#20924] IO#readline ignores the limit argument when the encoding is UTF-32LE and the limit would split a character javanthropus (Jeremy Bopp) via ruby-core
2024-11-29 16:35 ` [ruby-core:120059] " javanthropus (Jeremy Bopp) via ruby-core
2024-12-15  9:33 ` [ruby-core:120248] " nagachika (Tomoyuki Chikanaga) via ruby-core
2025-01-15  1:53 ` [ruby-core:120686] " k0kubun (Takashi Kokubun) via ruby-core

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).