zsh-workers
 help / color / mirror / code / Atom feed
From: "Daniel Shahaf" <d.s@daniel.shahaf.name>
To: zsh-workers@zsh.org
Subject: Writing XFail tests (was: Re: [BUG] POSIX arith: inf, nan should be variables)
Date: Wed, 01 Dec 2021 04:27:19 +0000	[thread overview]
Message-ID: <d4eab669-b726-4aa2-9fd4-d5f98f91a787@www.fastmail.com> (raw)
In-Reply-To: <CAH+w=7bCw2SgAjrVgATaQ+DKFFSrO_77JahB852NqP_zxa6CPQ@mail.gmail.com>

Bart Schaefer wrote on Wed, 01 Dec 2021 03:37 +00:00:
> On Tue, Nov 30, 2021 at 7:36 PM Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
>>
>> Maybe add a test based on the above code snippet?
>
> There's already an xfail test:
>
> Test ./E03posix.ztst was expected to fail, but passed.
> Was testing: All identifiers are variable references in POSIX arithmetic
>
> Just need to flip the state on that one.

I've taken the liberty of doing so.

Incidentally, when writing xfail tests I like to write them with minimal
expectations, so that the test will trigger as soon as zsh's behaviour
changes:
.
     what-is-2-plus-2
    -fD:Unit test for a future builtin
    *>[0-9]

Here, the test looks for something on stdout and is xfail, so it expects
no specific exit code and no specific stderr.  This way, the test will
flip as soon as stdout becomes correct, regardless of whether the exit
code and stderr are at that time also what the they are expected to be
in the end (usually 0 and empty, respectively).

The test also uses a pattern on stdout, so that the test will pass even if
there are, say, whitespace differences between the predicted output (at
the time the xfail test is written) and the actual output (later, when
the bug is fixed).  See workers/48916 for an example of this.

Once the bug is fixed, the test's expectations (all three of them:
stdout, stderr, exit code) can be changed to their permanent form:
.
     what-is-2-plus-2
    0:Unit test for what-is-2-plus-2
    >4

That is: I'd suggest to use «f» flag in conjunction with two of the
three flags [-dD].

Cheers,

Daniel


      reply	other threads:[~2021-12-01  4:28 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-15 17:40 [BUG] POSIX arith: inf, nan should be variables Martijn Dekker
2021-11-16  9:06 ` Oliver Kiddle
2021-11-16 12:55 ` Vincent Lefevre
2021-11-28 20:34   ` Oliver Kiddle
2021-12-01  3:31     ` Daniel Shahaf
2021-12-01  3:37       ` Bart Schaefer
2021-12-01  4:27         ` Daniel Shahaf [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=d4eab669-b726-4aa2-9fd4-d5f98f91a787@www.fastmail.com \
    --to=d.s@daniel.shahaf.name \
    --cc=zsh-workers@zsh.org \
    --subject='Writing XFail tests (was: Re: [BUG] POSIX arith: inf, nan should be variables)' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://git.vuxu.org/mirror/zsh/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).