[ruby-core:119207] [Ruby master Bug#20744] syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser

[ruby-core:119207] [Ruby master Bug#20744] syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser

[yahonda (Yasuo Honda) via ruby-core]

Issue #20744 has been reported by yahonda (Yasuo Honda).

----------------------------------------
Bug #20744: syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser
https://bugs.ruby-lang.org/issues/20744

* Author: yahonda (Yasuo Honda)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-09-15T01:06:11Z master 532af89e3b) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
I'm reporting this issue based on Rails Nightly CI against Ruby 3.4.0dev. 
https://buildkite.com/rails/rails-nightly/builds/1023#0191ed31-1d77-4705-ab65-cecab76e2c88/1175-1209
I have not able to create a minimum test case without Rails yet. 

### Steps to reproduce
```ruby
git clone https://github.com/rails/rails
cd rails/actionview
bin/test test/template/template_test.rb:278
```

### Expected behavior
It should pass as it Runs parser.y parser

```ruby
$ RUBYOPT=--parser=parse.y bin/test test/template/template_test.rb:278
/home/yahonda/.gem/ruby/3.4.0+0/gems/json-2.7.1/lib/json/common.rb:3: warning: ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
Running 36 tests in a single process (parallelization threshold is 50)
Run options: --seed 61197

# Running:

.

Finished in 0.037914s, 26.3756 runs/s, 79.1267 assertions/s.
1 runs, 3 assertions, 0 failures, 0 errors, 0 skips
$
```

### Actual behavior
It raises the following `syntax error found (SyntaxError)`

```ruby
$ bin/test test/template/template_test.rb:278
/home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'Kernel.require':
/home/yahonda/src/github.com/rails/rails/actionview/test/template/template_test.rb:278: syntax error found (SyntaxError)
  276 |       @template = new_template("<%# encoding: ISO-8859-1 %>\n<%# locals: (message: 'Hi!') %>\nhello \xFCmlat\n<%= message %>", virtual_path: nil)
  277 |       assert_equal Encoding::UTF_8, render.encoding
> 278 |       assert_match(/hello \u{fc}mlat\nHi!/, render)
      |                              ^~ UTF-8 mixed within US-ASCII source
  279 |     end
  280 |   end

	from /home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'block (2 levels) in Kernel#replace_require'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:62:in 'block in Rails::TestUnit::Runner.load_tests'
	from <internal:array>:42:in 'Array#each'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:60:in 'Rails::TestUnit::Runner.load_tests'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:52:in 'Rails::TestUnit::Runner.run'
	from /home/yahonda/src/github.com/rails/rails/tools/test.rb:18:in '<top (required)>'
	from bin/test:5:in 'Kernel#require_relative'
	from bin/test:5:in '<main>'
$
```




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:119211] [Ruby master Bug#20744] syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser

[ima1zumi (Mari Imaizumi) via ruby-core]

Issue #20744 has been updated by ima1zumi (Mari Imaizumi).


I'm unsure if this is the same case as Rails, but I found a reproduce code.

``` ruby
# encoding: US-ASCII

str = String.new("hello \xFC", encoding: Encoding::ISO_8859_1).encode(Encoding::UTF_8)
puts str =~ /hello \u{fc}/
```

### ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [arm64-darwin23]

```
0
```

### ruby 3.4.0dev (2024-09-15T01:06:11Z master 532af89e3b) +PRISM [arm64-darwin23]

```
iso88591.rb:
iso88591.rb:4: syntax error found (SyntaxError)
  2 |
  3 | str = String.new("hello \xFC", encoding: Encoding::ISO_8859_1).encode(Encoding::UTF_8)
> 4 | puts str =~ /hello \u{fc}/
    |                       ^~ UTF-8 mixed within US-ASCII source
  5 |
```


----------------------------------------
Bug #20744: syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser
https://bugs.ruby-lang.org/issues/20744#change-109791

* Author: yahonda (Yasuo Honda)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-09-15T01:06:11Z master 532af89e3b) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
I'm reporting this issue based on Rails Nightly CI against Ruby 3.4.0dev. 
https://buildkite.com/rails/rails-nightly/builds/1023#0191ed31-1d77-4705-ab65-cecab76e2c88/1175-1209
I have not able to create a minimum test case without Rails yet. 

### Steps to reproduce
```ruby
git clone https://github.com/rails/rails
cd rails/actionview
bin/test test/template/template_test.rb:278
```

### Expected behavior
It should pass as it Runs parser.y parser

```ruby
$ RUBYOPT=--parser=parse.y bin/test test/template/template_test.rb:278
/home/yahonda/.gem/ruby/3.4.0+0/gems/json-2.7.1/lib/json/common.rb:3: warning: ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
Running 36 tests in a single process (parallelization threshold is 50)
Run options: --seed 61197

# Running:

.

Finished in 0.037914s, 26.3756 runs/s, 79.1267 assertions/s.
1 runs, 3 assertions, 0 failures, 0 errors, 0 skips
$
```

### Actual behavior
It raises the following `syntax error found (SyntaxError)`

```ruby
$ bin/test test/template/template_test.rb:278
/home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'Kernel.require':
/home/yahonda/src/github.com/rails/rails/actionview/test/template/template_test.rb:278: syntax error found (SyntaxError)
  276 |       @template = new_template("<%# encoding: ISO-8859-1 %>\n<%# locals: (message: 'Hi!') %>\nhello \xFCmlat\n<%= message %>", virtual_path: nil)
  277 |       assert_equal Encoding::UTF_8, render.encoding
> 278 |       assert_match(/hello \u{fc}mlat\nHi!/, render)
      |                              ^~ UTF-8 mixed within US-ASCII source
  279 |     end
  280 |   end

	from /home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'block (2 levels) in Kernel#replace_require'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:62:in 'block in Rails::TestUnit::Runner.load_tests'
	from <internal:array>:42:in 'Array#each'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:60:in 'Rails::TestUnit::Runner.load_tests'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:52:in 'Rails::TestUnit::Runner.run'
	from /home/yahonda/src/github.com/rails/rails/tools/test.rb:18:in '<top (required)>'
	from bin/test:5:in 'Kernel#require_relative'
	from bin/test:5:in '<main>'
$
```




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:119218] [Ruby master Bug#20744] syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser

[kddnewton (Kevin Newton) via ruby-core]

Issue #20744 has been updated by kddnewton (Kevin Newton).

Assignee set to kddnewton (Kevin Newton)

Thank you for the report, I will take a look.

----------------------------------------
Bug #20744: syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser
https://bugs.ruby-lang.org/issues/20744#change-109797

* Author: yahonda (Yasuo Honda)
* Status: Open
* Assignee: kddnewton (Kevin Newton)
* ruby -v: ruby 3.4.0dev (2024-09-15T01:06:11Z master 532af89e3b) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
I'm reporting this issue based on Rails Nightly CI against Ruby 3.4.0dev. 
https://buildkite.com/rails/rails-nightly/builds/1023#0191ed31-1d77-4705-ab65-cecab76e2c88/1175-1209
I have not able to create a minimum test case without Rails yet. 

### Steps to reproduce
```ruby
git clone https://github.com/rails/rails
cd rails/actionview
bin/test test/template/template_test.rb:278
```

### Expected behavior
It should pass as it Runs parser.y parser

```ruby
$ RUBYOPT=--parser=parse.y bin/test test/template/template_test.rb:278
/home/yahonda/.gem/ruby/3.4.0+0/gems/json-2.7.1/lib/json/common.rb:3: warning: ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
Running 36 tests in a single process (parallelization threshold is 50)
Run options: --seed 61197

# Running:

.

Finished in 0.037914s, 26.3756 runs/s, 79.1267 assertions/s.
1 runs, 3 assertions, 0 failures, 0 errors, 0 skips
$
```

### Actual behavior
It raises the following `syntax error found (SyntaxError)`

```ruby
$ bin/test test/template/template_test.rb:278
/home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'Kernel.require':
/home/yahonda/src/github.com/rails/rails/actionview/test/template/template_test.rb:278: syntax error found (SyntaxError)
  276 |       @template = new_template("<%# encoding: ISO-8859-1 %>\n<%# locals: (message: 'Hi!') %>\nhello \xFCmlat\n<%= message %>", virtual_path: nil)
  277 |       assert_equal Encoding::UTF_8, render.encoding
> 278 |       assert_match(/hello \u{fc}mlat\nHi!/, render)
      |                              ^~ UTF-8 mixed within US-ASCII source
  279 |     end
  280 |   end

	from /home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'block (2 levels) in Kernel#replace_require'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:62:in 'block in Rails::TestUnit::Runner.load_tests'
	from <internal:array>:42:in 'Array#each'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:60:in 'Rails::TestUnit::Runner.load_tests'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:52:in 'Rails::TestUnit::Runner.run'
	from /home/yahonda/src/github.com/rails/rails/tools/test.rb:18:in '<top (required)>'
	from bin/test:5:in 'Kernel#require_relative'
	from bin/test:5:in '<main>'
$
```




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:119228] [Ruby master Bug#20744] syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser

[kddnewton (Kevin Newton) via ruby-core]

Issue #20744 has been updated by kddnewton (Kevin Newton).


I have opened a PR to fix this.

https://github.com/ruby/prism/pull/3071

----------------------------------------
Bug #20744: syntax error found (SyntaxError) `UTF-8 mixed within US-ASCII source` reported Ruby with Prism parser
https://bugs.ruby-lang.org/issues/20744#change-109808

* Author: yahonda (Yasuo Honda)
* Status: Open
* Assignee: kddnewton (Kevin Newton)
* ruby -v: ruby 3.4.0dev (2024-09-15T01:06:11Z master 532af89e3b) +PRISM [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
I'm reporting this issue based on Rails Nightly CI against Ruby 3.4.0dev. 
https://buildkite.com/rails/rails-nightly/builds/1023#0191ed31-1d77-4705-ab65-cecab76e2c88/1175-1209
I have not able to create a minimum test case without Rails yet. 

### Steps to reproduce
```ruby
git clone https://github.com/rails/rails
cd rails/actionview
bin/test test/template/template_test.rb:278
```

### Expected behavior
It should pass as it Runs parser.y parser

```ruby
$ RUBYOPT=--parser=parse.y bin/test test/template/template_test.rb:278
/home/yahonda/.gem/ruby/3.4.0+0/gems/json-2.7.1/lib/json/common.rb:3: warning: ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
Running 36 tests in a single process (parallelization threshold is 50)
Run options: --seed 61197

# Running:

.

Finished in 0.037914s, 26.3756 runs/s, 79.1267 assertions/s.
1 runs, 3 assertions, 0 failures, 0 errors, 0 skips
$
```

### Actual behavior
It raises the following `syntax error found (SyntaxError)`

```ruby
$ bin/test test/template/template_test.rb:278
/home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'Kernel.require':
/home/yahonda/src/github.com/rails/rails/actionview/test/template/template_test.rb:278: syntax error found (SyntaxError)
  276 |       @template = new_template("<%# encoding: ISO-8859-1 %>\n<%# locals: (message: 'Hi!') %>\nhello \xFCmlat\n<%= message %>", virtual_path: nil)
  277 |       assert_equal Encoding::UTF_8, render.encoding
> 278 |       assert_match(/hello \u{fc}mlat\nHi!/, render)
      |                              ^~ UTF-8 mixed within US-ASCII source
  279 |     end
  280 |   end

	from /home/yahonda/.rbenv/versions/trunk/lib/ruby/3.4.0+0/bundled_gems.rb:79:in 'block (2 levels) in Kernel#replace_require'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:62:in 'block in Rails::TestUnit::Runner.load_tests'
	from <internal:array>:42:in 'Array#each'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:60:in 'Rails::TestUnit::Runner.load_tests'
	from /home/yahonda/src/github.com/rails/rails/railties/lib/rails/test_unit/runner.rb:52:in 'Rails::TestUnit::Runner.run'
	from /home/yahonda/src/github.com/rails/rails/tools/test.rb:18:in '<top (required)>'
	from bin/test:5:in 'Kernel#require_relative'
	from bin/test:5:in '<main>'
$
```




-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/