[ruby-core:115512] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[ruby-core:115512] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[kddnewton (Kevin Newton) via ruby-core]

Issue #20025 has been reported by kddnewton (Kevin Newton).

----------------------------------------
Bug #20025: Parsing identifiers/constants is case-folding dependent
https://bugs.ruby-lang.org/issues/20025

* Author: kddnewton (Kevin Newton)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.

The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).

This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:

``` ruby
class Context < BasicObject
  def method_missing(name, *) = :identifier
  def self.const_missing(name) = :constant
end

encoding = Encoding::Windows_1253
character = 0xB5.chr(encoding)

source = "# encoding: #{encoding.name}\n#{character}\n"
result = Context.new.instance_eval(source)

puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"
puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"
puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"
puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"
puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"
puts "  parsed as #{result}"
```

this results in the output of:

```
Windows-1253 encoding of 0xB5
  [[:alpha:]] => true
  [[:alnum:]] => true
  [[:upper:]] => false
  [[:lower:]] => true
  parsed as constant
```

To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.

Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:115520] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[kddnewton (Kevin Newton) via ruby-core]

Issue #20025 has been updated by kddnewton (Kevin Newton).


I should additionally mention that this is the only codepoint in any encoding that this impacts. I ran a brute-force script to find any violations, please check me work here: https://gist.github.com/kddnewton/089d23d49adb5551792293fdb5bf64a0.

----------------------------------------
Bug #20025: Parsing identifiers/constants is case-folding dependent
https://bugs.ruby-lang.org/issues/20025#change-105449

* Author: kddnewton (Kevin Newton)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.

The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).

This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:

``` ruby
class Context < BasicObject
  def method_missing(name, *) = :identifier
  def self.const_missing(name) = :constant
end

encoding = Encoding::Windows_1253
character = 0xB5.chr(encoding)

source = "# encoding: #{encoding.name}\n#{character}\n"
result = Context.new.instance_eval(source)

puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"
puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"
puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"
puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"
puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"
puts "  parsed as #{result}"
```

this results in the output of:

```
Windows-1253 encoding of 0xB5
  [[:alpha:]] => true
  [[:alnum:]] => true
  [[:upper:]] => false
  [[:lower:]] => true
  parsed as constant
```

To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.

Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:115528] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[nobu (Nobuyoshi Nakada) via ruby-core]

Issue #20025 has been updated by nobu (Nobuyoshi Nakada).


https://github.com/ruby/ruby/pull/9059

----------------------------------------
Bug #20025: Parsing identifiers/constants is case-folding dependent
https://bugs.ruby-lang.org/issues/20025#change-105459

* Author: kddnewton (Kevin Newton)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.

The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).

This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:

``` ruby
class Context < BasicObject
  def method_missing(name, *) = :identifier
  def self.const_missing(name) = :constant
end

encoding = Encoding::Windows_1253
character = 0xB5.chr(encoding)

source = "# encoding: #{encoding.name}\n#{character}\n"
result = Context.new.instance_eval(source)

puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"
puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"
puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"
puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"
puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"
puts "  parsed as #{result}"
```

this results in the output of:

```
Windows-1253 encoding of 0xB5
  [[:alpha:]] => true
  [[:alnum:]] => true
  [[:upper:]] => false
  [[:lower:]] => true
  parsed as constant
```

To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.

Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:115529] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[nobu (Nobuyoshi Nakada) via ruby-core]

Issue #20025 has been updated by nobu (Nobuyoshi Nakada).


https://en.wikipedia.org/wiki/Windows-1253#cite_note-5
> This is in addition to the existing μ at 0xEC, which remains in place. Unicode calls the one at 0xB5 "micro sign" (U+00B5) and the one at 0xEC "Greek small letter Mu" (U+03BC), although the former is mapped to the latter by NFKC (although not NFC) Unicode normalization.

The reason is that micro sign is folded to small Mu in Windows-1253.

----------------------------------------
Bug #20025: Parsing identifiers/constants is case-folding dependent
https://bugs.ruby-lang.org/issues/20025#change-105460

* Author: kddnewton (Kevin Newton)
* Status: Open
* Priority: Normal
* Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN
----------------------------------------
When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.

The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).

This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:

``` ruby
class Context < BasicObject
  def method_missing(name, *) = :identifier
  def self.const_missing(name) = :constant
end

encoding = Encoding::Windows_1253
character = 0xB5.chr(encoding)

source = "# encoding: #{encoding.name}\n#{character}\n"
result = Context.new.instance_eval(source)

puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"
puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"
puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"
puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"
puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"
puts "  parsed as #{result}"
```

this results in the output of:

```
Windows-1253 encoding of 0xB5
  [[:alpha:]] => true
  [[:alnum:]] => true
  [[:upper:]] => false
  [[:lower:]] => true
  parsed as constant
```

To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.

Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:115584] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[duerst via ruby-core]

Issue #20025 has been updated by duerst (Martin Dürst).





@nobu (Nobuyoshi Nakada) wrote in #note-3:



> The reason is that micro sign is folded to small Mu in Windows-1253.



The micro sign is indeed folded to small mu in windows-1253. The reason is (most probably) that it is also folded this way in Unicode; see https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt. The actual data for this is the `'\354'` at https://github.com/ruby/ruby/blob/85bc80a51be0ceedcc57e7b6b779e6f8f885859e/enc/windows_1253.c#L67.



P.S.: I really feel like proposing to change all these octal constants to hexadecimal, in order to bring them into the current century and align them with all the other data surrounding character encoding. But I guess that should be a separate issue.







----------------------------------------

Bug #20025: Parsing identifiers/constants is case-folding dependent

https://bugs.ruby-lang.org/issues/20025#change-105516



* Author: kddnewton (Kevin Newton)

* Status: Closed

* Priority: Normal

* Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED

----------------------------------------

When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.



The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).



This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:



``` ruby

class Context < BasicObject

  def method_missing(name, *) = :identifier

  def self.const_missing(name) = :constant

end



encoding = Encoding::Windows_1253

character = 0xB5.chr(encoding)



source = "# encoding: #{encoding.name}\n#{character}\n"

result = Context.new.instance_eval(source)



puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"

puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"

puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"

puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"

puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"

puts "  parsed as #{result}"

```



this results in the output of:



```

Windows-1253 encoding of 0xB5

  [[:alpha:]] => true

  [[:alnum:]] => true

  [[:upper:]] => false

  [[:lower:]] => true

  parsed as constant

```



To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.



Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.







-- 

https://bugs.ruby-lang.org/

 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:121316] [Ruby master Bug#20025] Parsing identifiers/constants is case-folding dependent

[hsbt (Hiroshi SHIBATA) via ruby-core]

Issue #20025 has been updated by hsbt (Hiroshi SHIBATA).

Backport changed from 3.0: REQUIRED, 3.1: REQUIRED, 3.2: REQUIRED to 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONE

ruby_3_2 commit:6c24731837f88d67517cfc590cb496daed7a0ef5 merged revision(s) commit:79eb75a8dd64848f23e9efc465f06326b5d4b680.

----------------------------------------
Bug #20025: Parsing identifiers/constants is case-folding dependent
https://bugs.ruby-lang.org/issues/20025#change-112281

* Author: kddnewton (Kevin Newton)
* Status: Closed
* Backport: 3.0: REQUIRED, 3.1: REQUIRED, 3.2: DONE
----------------------------------------
When CRuby parses identifiers, it is encoding-dependent. Once the identifier is found, it determines if it starts with a uppercase or lowercase codepoint. This determines if the identifier is a constant or not.

The function is charge of this is `rb_sym_constant_char_p`. For non-unicode encodings where the leading byte has the top-bit set, this relies on onigmo's `mbc_case_fold` to determine if it is a constant or not (as opposed to `is_code_ctype`).

This works for almost every single codepoint in every encoding, but has one very weird edge case. In the Windows-1253 encoding for the 0xB5 byte, it's the micro sign. The micro sign, when case folded, becomes the uppercase mu character, and then the lowercase mu character, or 0xEC. This means that even though 0xB5 reports itself as being a lowercase codepoint, it gets parsed as a constant. This example might make this more clear:

``` ruby
class Context < BasicObject
  def method_missing(name, *) = :identifier
  def self.const_missing(name) = :constant
end

encoding = Encoding::Windows_1253
character = 0xB5.chr(encoding)

source = "# encoding: #{encoding.name}\n#{character}\n"
result = Context.new.instance_eval(source)

puts "#{encoding.name} encoding of 0x#{character.ord.to_s(16).upcase}"
puts "  [[:alpha:]] => #{character.match?(/[[:alpha:]]/)}"
puts "  [[:alnum:]] => #{character.match?(/[[:alnum:]]/)}"
puts "  [[:upper:]] => #{character.match?(/[[:upper:]]/)}"
puts "  [[:lower:]] => #{character.match?(/[[:lower:]]/)}"
puts "  parsed as #{result}"
```

this results in the output of:

```
Windows-1253 encoding of 0xB5
  [[:alpha:]] => true
  [[:alnum:]] => true
  [[:upper:]] => false
  [[:lower:]] => true
  parsed as constant
```

To be clear, I don't think the case-folding is incorrect here (and @duerst confirms that it is correct). I believe instead that it is incorrect to use case-folding here to determine if a codepoint is uppercase or not.

Note that this only impacts this one codepoint in this one encoding, so I don't believe this is actually a large-scale problem. But I found it surprising, and think we should change it.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/