A suggestion RE Xfail tests

A suggestion RE Xfail tests

[Bart Schaefer]

It would be helpful if tests that are expected to fail could list both
the expected "correct" result (that is, the result if the test were to
succeed, which would cause the Xfail test to fail) and the expected
"wrong" output ... so that if the way a test goes wrong changes, that
too can be seen.

Yes, this can be done now by repeating the test twice, once as an
xfail and once as a regular test but with the "broken" result ... but
it might be easier to keep track of if it were all in one place.

(No action on this expected pre-5.9-release, just musing.)

Re: A suggestion RE Xfail tests

[Daniel Shahaf]

Bart Schaefer wrote on Wed, 04 May 2022 01:44 +00:00:
> It would be helpful if tests that are expected to fail could list both
> the expected "correct" result (that is, the result if the test were to
> succeed, which would cause the Xfail test to fail) and the expected
> "wrong" output ... so that if the way a test goes wrong changes, that
> too can be seen.
>

Fair request.

FWIW, I don't know of any other test harness that supports this.

> Yes, this can be done now by repeating the test twice, once as an
> xfail and once as a regular test but with the "broken" result ... but
> it might be easier to keep track of if it were all in one place.
>

How about if, for the time being, we use the "duplicate the test"
approach, with F: lines or comments as needed?  Once we have some
experience with that, we may have a better idea of what test harness
functionality to implement.

> (No action on this expected pre-5.9-release, just musing.)