From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 13113 invoked from network); 1 Dec 2021 04:28:18 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 1 Dec 2021 04:28:18 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1638332899; b=rWJB5kptpZnxUaWpPGtQuvd4rJD4wvrbR/INVx/1sYH6UVJGgFH7fJL0efI/Epc0598syLHxrT TQ6PW4H7DBsPCY82lZVe4SHS1RxnZ8zZz2dVJF0BBcjqDYa8PTHt4XbjUoXntjLWrDxO1cEwuW uS2/gZvkAU1IU2XdwpkFlx+BU7Cn87ajKhkaXTS0HABUBY9neoN51BJFD7etgbXx+ey7N97lOx tDKF0BOws/pF934Agsmj/5RixewVx7vqr+ZqeYaBcANqWBXZr0A44HgdUGlruQYgNYSop3kfuZ J4xT1HFSDjhgu1F6qiuiMCy12d8PQC5X7mGkRMtN47Nc1A==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (out5-smtp.messagingengine.com) smtp.remote-ip=66.111.4.29; dkim=pass header.d=daniel.shahaf.name header.s=fm2 header.a=rsa-sha256; dkim=pass header.d=messagingengine.com header.s=fm1 header.a=rsa-sha256; dmarc=none header.from=daniel.shahaf.name; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1638332899; bh=O2Fo5RoZo/BCGxsbXbIj6F9AORLj/mX48Bk+nQem7hw=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Transfer-Encoding:Content-Type:Subject:To:From:Date: References:In-Reply-To:Message-ID:MIME-Version:DKIM-Signature: DKIM-Signature:DKIM-Signature; b=qt65ID/hqTDgokjcq/EQVSbX6kTW+PqHJvzcTGcMR/FmbAsOPWTP8kvxK70a2qfYoPkuuyQvvK VeYYeJOh2SkRtemZhqFU8+nxjofIUGLUNwsep4h759b6M/9Ae3q03xx5o/PS3cUSsScHS/eAPb CB0AeN+xhALHHtjAchcEeeEYg3tdhTIx3m9Mkq9kJ6Np7+mPvIDr42QswrCxz2bogOsO8ShiFt dF9TOuqo3LG3AahPbOEDDSNYv1SllHOWgn/XCS7dTIpf6lACop5hyBiiZ0dqvFtQZJUDdKl7ud d1EbPipbxI0+Icd1HDLDIIsat/RzmV0ZRVz9YfsPkQgCxg==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Transfer-Encoding: Content-Type:Subject:To:From:Date:References:In-Reply-To:Message-Id: Mime-Version:Reply-To:Cc:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=rJr315ndjPykNaryIL/CHYucq8EtsrOmQhmHwhpzDxQ=; b=pgr+E6E2eBj8iLiOkVCuLynAeS pytejzHZKoWQRzkf+ahwET2Wwr3st0Kd71+asf6lm7w/kIFwXdPO04eBKxOS53u5gWAGkUXifVWc7 +Xzz2tgnZmRMV0vx8dfEeGa0Nqp7VV4+A6MSsdVIyZI0V/fCMMhiki7fI1XNo1odTJ30N3GsonEn6 L7Xrp577Og4hVNsuaIZI0W8TwlbjXku36ONBCZnucjTyyiGoBwUwGHy4EnGEFHHtsA3eAZgqdF0BD mJ1d5SUIImrzQ3fyYWxoHr9lUC3EevxZoeKblmFWM1kHUMinjqwglyqEIua8QIOuLOLNukeLPzQpI HPbib4gA==; Received: from authenticated user by zero.zsh.org with local id 1msHEA-00048w-28; Wed, 01 Dec 2021 04:28:18 +0000 Authentication-Results: zsh.org; iprev=pass (out5-smtp.messagingengine.com) smtp.remote-ip=66.111.4.29; dkim=pass header.d=daniel.shahaf.name header.s=fm2 header.a=rsa-sha256; dkim=pass header.d=messagingengine.com header.s=fm1 header.a=rsa-sha256; dmarc=none header.from=daniel.shahaf.name; arc=none Received: from out5-smtp.messagingengine.com ([66.111.4.29]:39111) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_256_GCM_SHA384:256) id 1msHDZ-0003rV-CI; Wed, 01 Dec 2021 04:27:42 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id E478D5C022F for ; Tue, 30 Nov 2021 23:27:39 -0500 (EST) Received: from imap46 ([10.202.2.96]) by compute5.internal (MEProxy); Tue, 30 Nov 2021 23:27:39 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= daniel.shahaf.name; h=mime-version:message-id:in-reply-to :references:date:from:to:subject:content-type :content-transfer-encoding; s=fm2; bh=rJr315ndjPykNaryIL/CHYucq8 EtsrOmQhmHwhpzDxQ=; b=iKtimILGA5UxUT+//JMpAUN56K8zz1EOy1qpB3/OU2 zPRJHipGQ+FFj5tLDoJwTHV3Q1HZJcucoPmKroTxXMWqP78AWxU8g9XyaWENqAMP Akt56V0/nU2p+376Lz5RYKAZsF9JDRBIxFSY1rjdLa7w2vj4NYRj8RYy8i0tm6ik plAY9JfXynSMBNEm07WeUIRAECsktRO7XBeGhYxsBcZ37O9eaotmQuay4FoQUtsI SmeNhN1QX1qNH6ZlSWafsmQhkDrxlcVWi21julx61wp771aemUhq76Vzn4G90D6a k7gxn+E7UCpOtte4qOLwW3d7QXg/8Uq+09ASjB6Qtx4w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=rJr315ndjPykNaryIL/CHYucq8EtsrOmQhmHwhpzD xQ=; b=COMW+kAejUnKm/7OnmxVVGlBEgCHKkW2I/k0lPsxmNRDNJqO9H12p0pTG SBofs3sre9W90KSrFe2IIQ1/aqzNav+FiF60MJ217WFnOHnBCFGUn0WnRii8mG5r sMjlJAUKDlrAEC18z4lAiMXu/cTEv7npDKWC/Ir+VqdcEo/9vOlYAEH1mo/3D4z6 6EoCPyGIlqPrgRS96ZGCQW04u8yQZFTBvBl/qBoR/KSATHI7Ey/03/agEqbUgViy saXngg8rSASk5Je4W0fRbvqF0P7RGJP707YUmePU+2aSgqnTjn334cjaVrueC6Oq aqxwToYQhqgxxBAbvYY9nIvU7XhJQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddriedvgdejudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgfgsehtqh ertderreejnecuhfhrohhmpedfffgrnhhivghlucfuhhgrhhgrfhdfuceougdrshesuggr nhhivghlrdhshhgrhhgrfhdrnhgrmhgvqeenucggtffrrghtthgvrhhnpeefhfekfeevie elleehiefftdekudejledvhfdtjedtgeegvdeuffefleehhfdvgfenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegurdhssegurghnihgvlhdrsh hhrghhrghfrdhnrghmvg X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id BC7A51EE007B; Tue, 30 Nov 2021 23:27:39 -0500 (EST) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.5.0-alpha0-4458-g51a91c06b2-fm-20211130.004-g51a91c06 Mime-Version: 1.0 Message-Id: In-Reply-To: References: <20211116125522.GA121624@zira.vinc17.org> <53575-1638131681.212237@WIRp.dqll.oMb9> <20211201033100.GC10648@tarpaulin.shahaf.local2> Date: Wed, 01 Dec 2021 04:27:19 +0000 From: "Daniel Shahaf" To: zsh-workers@zsh.org Subject: Writing XFail tests (was: Re: [BUG] POSIX arith: inf, nan should be variables) Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Seq: 49626 Archived-At: X-Loop: zsh-workers@zsh.org Errors-To: zsh-workers-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-workers-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: Bart Schaefer wrote on Wed, 01 Dec 2021 03:37 +00:00: > On Tue, Nov 30, 2021 at 7:36 PM Daniel Shahaf = wrote: >> >> Maybe add a test based on the above code snippet? > > There's already an xfail test: > > Test ./E03posix.ztst was expected to fail, but passed. > Was testing: All identifiers are variable references in POSIX arithmet= ic > > Just need to flip the state on that one. I've taken the liberty of doing so. Incidentally, when writing xfail tests I like to write them with minimal expectations, so that the test will trigger as soon as zsh's behaviour changes: . what-is-2-plus-2 -fD:Unit test for a future builtin *>[0-9] Here, the test looks for something on stdout and is xfail, so it expects no specific exit code and no specific stderr. This way, the test will flip as soon as stdout becomes correct, regardless of whether the exit code and stderr are at that time also what the they are expected to be in the end (usually 0 and empty, respectively). The test also uses a pattern on stdout, so that the test will pass even = if there are, say, whitespace differences between the predicted output (at the time the xfail test is written) and the actual output (later, when the bug is fixed). See workers/48916 for an example of this. Once the bug is fixed, the test's expectations (all three of them: stdout, stderr, exit code) can be changed to their permanent form: . what-is-2-plus-2 0:Unit test for what-is-2-plus-2 >4 That is: I'd suggest to use =C2=ABf=C2=BB flag in conjunction with two o= f the three flags [-dD]. Cheers, Daniel