[ruby-core:121201] [Ruby master Bug#21162] Regexp casefold mismatch for latin1 supplemental chars

[ruby-core:121201] [Ruby master Bug#21162] Regexp casefold mismatch for latin1 supplemental chars

[headius (Charles Nutter) via ruby-core]

Issue #21162 has been reported by headius (Charles Nutter).

----------------------------------------
Bug #21162: Regexp casefold mismatch for latin1 supplemental chars
https://bugs.ruby-lang.org/issues/21162

* Author: headius (Charles Nutter)
* Status: Open
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN
----------------------------------------
Originally reported to joni repo with a possible fix here: https://github.com/jruby/joni/pull/20

>From that PR:

> When a character is less than or equal to single byte size (0xff),
> yet it takes more than 1 byte in the current encoding, the
> case folding code incorrectly put it in bitset instead of code
> range. As a result, for utf8 encoding, casefold works incorrectly
> on characters in range \u0080 to \u00ff (latin1 supplement).
> 
> Before fix:
> 
> "\u00c2" [\u00e0-\u00e5] returns false
> "\u00c2" [\u00e2] returns false
> "\u00c2" \u00e2 returns true

As a Ruby example:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin24]
nil
```

The PR above was rebased in https://github.com/jruby/joni/pull/85. When that patch is incorporated into JRuby, it behaves as expected:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
jruby 10.0.0.0-SNAPSHOT (3.4.0) 2025-02-27 c0e5008419 OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]
0
```

This bug may affect other casefold situations. As I am not very familiar with this code in CRuby (other than from maintaining the port in joni) I would like others to evaluate this fix and help find other places it is needed.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:121204] [Ruby master Bug#21162] Regexp casefold mismatch for latin1 supplemental chars

[nobu (Nobuyoshi Nakada) via ruby-core]

Issue #21162 has been updated by nobu (Nobuyoshi Nakada).


Sounds like same as #16145.

----------------------------------------
Bug #21162: Regexp casefold mismatch for latin1 supplemental chars
https://bugs.ruby-lang.org/issues/21162#change-112145

* Author: headius (Charles Nutter)
* Status: Open
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN
----------------------------------------
Originally reported to joni repo with a possible fix here: https://github.com/jruby/joni/pull/20

>From that PR:

> When a character is less than or equal to single byte size (0xff),
> yet it takes more than 1 byte in the current encoding, the
> case folding code incorrectly put it in bitset instead of code
> range. As a result, for utf8 encoding, casefold works incorrectly
> on characters in range \u0080 to \u00ff (latin1 supplement).
> 
> Before fix:
> 
> "\u00c2" [\u00e0-\u00e5] returns false
> "\u00c2" [\u00e2] returns false
> "\u00c2" \u00e2 returns true

As a Ruby example:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin24]
nil
```

The PR above was rebased in https://github.com/jruby/joni/pull/85. When that patch is incorporated into JRuby, it behaves as expected:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
jruby 10.0.0.0-SNAPSHOT (3.4.0) 2025-02-27 c0e5008419 OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]
0
```

This bug may affect other casefold situations. As I am not very familiar with this code in CRuby (other than from maintaining the port in joni) I would like others to evaluate this fix and help find other places it is needed.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:121225] [Ruby master Bug#21162] Regexp casefold mismatch for latin1 supplemental chars

[headius (Charles Nutter) via ruby-core]

Issue #21162 has been updated by headius (Charles Nutter).


@nobu Certainly could be and the fix looks similar.

@mjrzasa What do you think? Does your fix repair this problem?

----------------------------------------
Bug #21162: Regexp casefold mismatch for latin1 supplemental chars
https://bugs.ruby-lang.org/issues/21162#change-112167

* Author: headius (Charles Nutter)
* Status: Open
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN
----------------------------------------
Originally reported to joni repo with a possible fix here: https://github.com/jruby/joni/pull/20

>From that PR:

> When a character is less than or equal to single byte size (0xff),
> yet it takes more than 1 byte in the current encoding, the
> case folding code incorrectly put it in bitset instead of code
> range. As a result, for utf8 encoding, casefold works incorrectly
> on characters in range \u0080 to \u00ff (latin1 supplement).
> 
> Before fix:
> 
> "\u00c2" [\u00e0-\u00e5] returns false
> "\u00c2" [\u00e2] returns false
> "\u00c2" \u00e2 returns true

As a Ruby example:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin24]
nil
```

The PR above was rebased in https://github.com/jruby/joni/pull/85. When that patch is incorporated into JRuby, it behaves as expected:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
jruby 10.0.0.0-SNAPSHOT (3.4.0) 2025-02-27 c0e5008419 OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]
0
```

This bug may affect other casefold situations. As I am not very familiar with this code in CRuby (other than from maintaining the port in joni) I would like others to evaluate this fix and help find other places it is needed.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:121255] [Ruby master Bug#21162] Regexp casefold mismatch for latin1 supplemental chars

[nobu (Nobuyoshi Nakada) via ruby-core]

Issue #21162 has been updated by nobu (Nobuyoshi Nakada).

Status changed from Open to Closed

I

----------------------------------------
Bug #21162: Regexp casefold mismatch for latin1 supplemental chars
https://bugs.ruby-lang.org/issues/21162#change-112208

* Author: headius (Charles Nutter)
* Status: Closed
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN
----------------------------------------
Originally reported to joni repo with a possible fix here: https://github.com/jruby/joni/pull/20

>From that PR:

> When a character is less than or equal to single byte size (0xff),
> yet it takes more than 1 byte in the current encoding, the
> case folding code incorrectly put it in bitset instead of code
> range. As a result, for utf8 encoding, casefold works incorrectly
> on characters in range \u0080 to \u00ff (latin1 supplement).
> 
> Before fix:
> 
> "\u00c2" [\u00e0-\u00e5] returns false
> "\u00c2" [\u00e2] returns false
> "\u00c2" \u00e2 returns true

As a Ruby example:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin24]
nil
```

The PR above was rebased in https://github.com/jruby/joni/pull/85. When that patch is incorporated into JRuby, it behaves as expected:

```
$ ruby -v -e 'p(/[\u00e0-\u00e5]/i =~ "\u00c2")'
jruby 10.0.0.0-SNAPSHOT (3.4.0) 2025-02-27 c0e5008419 OpenJDK 64-Bit Server VM 21.0.5+11-LTS on 21.0.5+11-LTS +indy +jit [arm64-darwin]
0
```

This bug may affect other casefold situations. As I am not very familiar with this code in CRuby (other than from maintaining the port in joni) I would like others to evaluate this fix and help find other places it is needed.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/