From: "Daniel Shahaf" <email@example.com> To: firstname.lastname@example.org Subject: Writing XFail tests (was: Re: [BUG] POSIX arith: inf, nan should be variables) Date: Wed, 01 Dec 2021 04:27:19 +0000 [thread overview] Message-ID: <email@example.com> (raw) In-Reply-To: <CAH+w=7bCw2SgAjrVgATaQ+DKFFSrO_77JahB852NqP_zxa6CPQ@mail.gmail.com> Bart Schaefer wrote on Wed, 01 Dec 2021 03:37 +00:00: > On Tue, Nov 30, 2021 at 7:36 PM Daniel Shahaf <firstname.lastname@example.org> wrote: >> >> Maybe add a test based on the above code snippet? > > There's already an xfail test: > > Test ./E03posix.ztst was expected to fail, but passed. > Was testing: All identifiers are variable references in POSIX arithmetic > > Just need to flip the state on that one. I've taken the liberty of doing so. Incidentally, when writing xfail tests I like to write them with minimal expectations, so that the test will trigger as soon as zsh's behaviour changes: . what-is-2-plus-2 -fD:Unit test for a future builtin *>[0-9] Here, the test looks for something on stdout and is xfail, so it expects no specific exit code and no specific stderr. This way, the test will flip as soon as stdout becomes correct, regardless of whether the exit code and stderr are at that time also what the they are expected to be in the end (usually 0 and empty, respectively). The test also uses a pattern on stdout, so that the test will pass even if there are, say, whitespace differences between the predicted output (at the time the xfail test is written) and the actual output (later, when the bug is fixed). See workers/48916 for an example of this. Once the bug is fixed, the test's expectations (all three of them: stdout, stderr, exit code) can be changed to their permanent form: . what-is-2-plus-2 0:Unit test for what-is-2-plus-2 >4 That is: I'd suggest to use «f» flag in conjunction with two of the three flags [-dD]. Cheers, Daniel
prev parent reply other threads:[~2021-12-01 4:28 UTC|newest] Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-11-15 17:40 [BUG] POSIX arith: inf, nan should be variables Martijn Dekker 2021-11-16 9:06 ` Oliver Kiddle 2021-11-16 12:55 ` Vincent Lefevre 2021-11-28 20:34 ` Oliver Kiddle 2021-12-01 3:31 ` Daniel Shahaf 2021-12-01 3:37 ` Bart Schaefer 2021-12-01 4:27 ` Daniel Shahaf [this message]
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --email@example.com \ --firstname.lastname@example.org \ --email@example.com \ --subject='Writing XFail tests (was: Re: [BUG] POSIX arith: inf, nan should be variables)' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
Code repositories for project(s) associated with this inbox: https://git.vuxu.org/mirror/zsh/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).