[ruby-core:119175] [Ruby master Misc#20739] Test suite does not carry over CLI options

[ruby-core:119175] [Ruby master Misc#20739] Test suite does not carry over CLI options

[kddnewton (Kevin Newton) via ruby-core]

Issue #20739 has been reported by kddnewton (Kevin Newton).

----------------------------------------
Misc #20739: Test suite does not carry over CLI options
https://bugs.ruby-lang.org/issues/20739

* Author: kddnewton (Kevin Newton)
* Status: Open
----------------------------------------
I'm not sure if anything needs to change here, but I wanted to report this in case anyone had any particular ideas. When running the ruby/ruby test suite, lots of tests call `EnvUtil.invoke_ruby` directly or indirectly through `assert_in_out_err`. That spawns a subprocess that runs Ruby and reports its status back through status, stdout, and stderr. The issue is, the subprocess does not always use the same options, which means there are lots of different ways it can be run.

For RJIT/YJIT, we have `-DRJIT_FORCE_ENABLE` and `-DYJIT_FORCE_ENABLE`, which forces the subprocess to have it enabled. Still, this isn't always used, so when running the test suite for RJIT/YJIT, a lot of tests are not actually running with those compilers.

Prism and parse.y have a similar problem. When running in the same process, it will respect the `--parser` option. But when running in a subprocess it will fall back to whatever the default parser is. This presents a problem for some tests that assert against RUBY_DESCRIPTION in a subprocess, as it may have `+PRISM` in the description.

We have tried to fix it before by adding RUN_OPTS/RUBYOPT to the subprocess options, but this causes lots of other problems. I am not sure of the correct solution, or if we should even bother fixing this. Please let me know what you think.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/

[ruby-core:119183] [Ruby master Misc#20739] Test suite does not carry over CLI options

[Eregon (Benoit Daloze) via ruby-core]

Issue #20739 has been updated by Eregon (Benoit Daloze).


I think it's a real issue and sometimes valuable to fix.
#6648 could be helpful to automatically inherit passed flags.
Alternatively, we could ensure we only pass options for tests through `RUBYOPT`, and that RUBYOPT is never cleared/overridden without considering the previous value.

MSpec currently handles this by having an extra subprocess, so all test processes always have all the `-T-flag` options consistently.
(#6648 would be nicer)

----------------------------------------
Misc #20739: Test suite does not carry over CLI options
https://bugs.ruby-lang.org/issues/20739#change-109763

* Author: kddnewton (Kevin Newton)
* Status: Open
----------------------------------------
I'm not sure if anything needs to change here, but I wanted to report this in case anyone had any particular ideas. When running the ruby/ruby test suite, lots of tests call `EnvUtil.invoke_ruby` directly or indirectly through `assert_in_out_err`. That spawns a subprocess that runs Ruby and reports its status back through status, stdout, and stderr. The issue is, the subprocess does not always use the same options, which means there are lots of different ways it can be run.

For RJIT/YJIT, we have `-DRJIT_FORCE_ENABLE` and `-DYJIT_FORCE_ENABLE`, which forces the subprocess to have it enabled. Still, this isn't always used, so when running the test suite for RJIT/YJIT, a lot of tests are not actually running with those compilers.

Prism and parse.y have a similar problem. When running in the same process, it will respect the `--parser` option. But when running in a subprocess it will fall back to whatever the default parser is. This presents a problem for some tests that assert against RUBY_DESCRIPTION in a subprocess, as it may have `+PRISM` in the description.

We have tried to fix it before by adding RUN_OPTS/RUBYOPT to the subprocess options, but this causes lots of other problems. I am not sure of the correct solution, or if we should even bother fixing this. Please let me know what you think.



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/lists/ruby-core.ml.ruby-lang.org/