Writing XFail tests (was: Re: [BUG] POSIX arith: inf, nan should be variables)

["Daniel Shahaf" <d.s@daniel.shahaf.name>]

Bart Schaefer wrote on Wed, 01 Dec 2021 03:37 +00:00:
> On Tue, Nov 30, 2021 at 7:36 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>>
>> Maybe add a test based on the above code snippet?
>
> There's already an xfail test:
>
> Test ./E03posix.ztst was expected to fail, but passed.
> Was testing: All identifiers are variable references in POSIX arithmetic
>
> Just need to flip the state on that one.

I've taken the liberty of doing so.

Incidentally, when writing xfail tests I like to write them with minimal
expectations, so that the test will trigger as soon as zsh's behaviour
changes:
.
     what-is-2-plus-2
    -fD:Unit test for a future builtin
    *>[0-9]

Here, the test looks for something on stdout and is xfail, so it expects
no specific exit code and no specific stderr.  This way, the test will
flip as soon as stdout becomes correct, regardless of whether the exit
code and stderr are at that time also what the they are expected to be
in the end (usually 0 and empty, respectively).

The test also uses a pattern on stdout, so that the test will pass even if
there are, say, whitespace differences between the predicted output (at
the time the xfail test is written) and the actual output (later, when
the bug is fixed).  See workers/48916 for an example of this.

Once the bug is fixed, the test's expectations (all three of them:
stdout, stderr, exit code) can be changed to their permanent form:
.
     what-is-2-plus-2
    0:Unit test for what-is-2-plus-2
    >4

That is: I'd suggest to use «f» flag in conjunction with two of the
three flags [-dD].

Cheers,

Daniel