From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 26901 invoked from network); 4 May 2022 02:52:59 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 4 May 2022 02:52:59 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1651632779; b=G62zWd1fckHO4pbF7Ee5vdSlSv3B7wxJFFv0pEq1fRQ7M1C+EpojLDclJX5Hk/nYKgki9B3tdN 84TgJ5OXbpmx2Mf6vugmDjPvZm24z8XJ0VXLviPkS8L9OhvNwe6b+51ECvF12+BuyxWCBj4zBg WS1aHTj906j9N1b5ni0P6twKIapJkCIogxb/Jsl9vEWzy2STfxe/wUxz5GnVB9VljPhApDlV5v GragtqAs/nld36uXHmje/QLojfi6Uol7udMj6Ko8/PrGA7GLafr2TRjtIIhiTSNKA9ZdhUzfQZ 4yCOTZuuOXpp2B7guxwbnt+5Rqb3Q9GbIRfJ9n5GMaVs4g==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (out4-smtp.messagingengine.com) smtp.remote-ip=66.111.4.28; dkim=pass header.d=daniel.shahaf.name header.s=fm3 header.a=rsa-sha256; dkim=pass header.d=messagingengine.com header.s=fm1 header.a=rsa-sha256; dmarc=none header.from=daniel.shahaf.name; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1651632779; bh=1KH1cTPPub+7lJWf/husnaRJHPJMEtd9/Y6EGjVwleQ=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:Content-Type:Subject:To:From:Date:References:In-Reply-To: Message-ID:MIME-Version:DKIM-Signature:DKIM-Signature:DKIM-Signature; b=fJNkuJ6rz3QIJMwCC0qEd1dMfX0qubXJqiNpl3QmAW5u3ZHO7XdQfpaF+OTJTHOpS7p/sgtLWW tpAFJCsM1c3/W7Zg1ZFkziNb/o83dqY03FgyoczVcwbCjgMhYOW3OInitdtRk1OvmTHjCeq27K SLuWL+WFMuS4GLiO4nyQBlQfEOfQxT6x6vag02KsQ+37iXJ1DKP+YzCamPfI4bztsPclrt1XgP d58i4ZdmYXNBI6Sj7h8QUl8PzF2X4qlP/VhV/DtYyq8Wn5wJduMUbqILYFQJLB0MupNnXhJshF DBDjlPs5VfnMO4w5VJ6Od+V6lHJbCynjolfAdL4P571FLg==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:Content-Type:Subject:To:From:Date: References:In-Reply-To:Message-Id:Mime-Version:Reply-To:Cc: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=8s7uZaENfgYxqJiwInvcbQfoBoXVTyKXJxrKE6fO3D0=; b=QlOjulQ9fBAtu1LcnhZkQIdnj1 WFTqv/d4sOR4RCzTP7ib2UUzTyIm8Gq4YElr9WTtcKIzKW2xNrV3UN3oWixbMAnW4J+ibNO7j3G2n yKE3z7JI0fq31uV46+Z0ljd6X2zQTTFwp/qkpMxPKOYYN9sclDrNc7anCpPucU7ybBcvu2ZKlLUrp GpPrymwABEHIu8teNgQZavMEyiL7v0wEj3LIMAsFjN1PtIyb6xHT/ciFOcINAd5DG0felCdLwSZNd hJktMD93/OJiBWH9/WG5q/6KFiCYOUor8qlEIGpJuCVXk0H4kV7GB1Tv3DOeMW6H120VRAy6C4Xt7 urHd7JCA==; Received: from authenticated user by zero.zsh.org with local id 1nm58M-000HbF-LI; Wed, 04 May 2022 02:52:58 +0000 Authentication-Results: zsh.org; iprev=pass (out4-smtp.messagingengine.com) smtp.remote-ip=66.111.4.28; dkim=pass header.d=daniel.shahaf.name header.s=fm3 header.a=rsa-sha256; dkim=pass header.d=messagingengine.com header.s=fm1 header.a=rsa-sha256; dmarc=none header.from=daniel.shahaf.name; arc=none Received: from out4-smtp.messagingengine.com ([66.111.4.28]:60673) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_256_GCM_SHA384:256) id 1nm57n-000HHW-LF; Wed, 04 May 2022 02:52:25 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 4296D5C0185 for ; Tue, 3 May 2022 22:52:22 -0400 (EDT) Received: from imap46 ([10.202.2.96]) by compute2.internal (MEProxy); Tue, 03 May 2022 22:52:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= daniel.shahaf.name; h=cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm3; t=1651632742; x= 1651719142; bh=8s7uZaENfgYxqJiwInvcbQfoBoXVTyKXJxrKE6fO3D0=; b=E nj/iIDMHJXtJqp46F8ikzhGHUIYIAYvjEv2zmOpYyMxr8BbZ31vRfjnydctzqUf6 rEI3MX3Xxd52ynys9gbfV9WVrQQNnw7UiZTw/9jut+6nePgWUBvJcV+l9XQt1nTa Zr8RfVv4QAlGU6M48IWT4zo9rjAQbcxGibI1p59w9dT8g+3vyIJwUFC2xSBRv2o+ wFYTfHTtMcAZ/dfcddmvOCK+i8x0rJPfEECPBOdXeGsuCwsPT2nslk2XG4oc5DV0 AHxCrs+UF3r35ZylcXxmGl8lVwiQyywMjSxujG+Z/w1743ZzTyObRX3MulTWBihC rlnJX58iQP4xuMS+DaVvw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1651632742; x= 1651719142; bh=8s7uZaENfgYxqJiwInvcbQfoBoXVTyKXJxrKE6fO3D0=; b=Q HI9B1HRz9BMBqtc0ox/4NdVwWupPjj9WCL/vawXQ4aoRSm0eJOKgd/p9JLY2sYOE QS1cg0QSq+vzFCCoTRHid3ioFFtWC3mHUvayyu1H/EW3IhefY9A9enz12H/cPFlB D/rEg9TT21/uI/iL1AdaQBVIXffsQ3Y35eKIhsaCVTfZGIsJNjD56SEfDhc1nV74 dP1Ed4Lht52LY34/NczkJfhP+Ok9xzyh8sxF4xBf0yqIkR21aQLfhClxOi6JhGVp ZFqKizRl9MdLSNKwmwtg0wwjCNFWNJkEVkl5EtTxm86BHEx/IlA1dJNla3CINMbL HJBItAYiFNTOqx6SyQOeg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdekgdeigecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepofgfggfkjghffffhvffutgesthdtre dtreertdenucfhrhhomhepfdffrghnihgvlhcuufhhrghhrghffdcuoegurdhssegurghn ihgvlhdrshhhrghhrghfrdhnrghmvgeqnecuggftrfgrthhtvghrnhepjeeuveehueekje dtffegveelhfethefhlefhfffgudeuhfdtvdehhfejkeeghedtnecuvehluhhsthgvrhfu ihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepugdrshesuggrnhhivghlrdhshh grhhgrfhdrnhgrmhgv X-ME-Proxy: Received: by mailuser.nyi.internal (Postfix, from userid 501) id 088AD2A2005D; Tue, 3 May 2022 22:52:21 -0400 (EDT) X-Mailer: MessagingEngine.com Webmail Interface User-Agent: Cyrus-JMAP/3.7.0-alpha0-591-gfe6c3a2700-fm-20220427.001-gfe6c3a27 Mime-Version: 1.0 Message-Id: <6ae076b1-6f0d-40d8-b1f5-1470048f2332@www.fastmail.com> In-Reply-To: References: Date: Wed, 04 May 2022 02:52:01 +0000 From: "Daniel Shahaf" To: "Zsh hackers list" Subject: Re: A suggestion RE Xfail tests Content-Type: text/plain X-Seq: 50173 Archived-At: X-Loop: zsh-workers@zsh.org Errors-To: zsh-workers-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-workers-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: Bart Schaefer wrote on Wed, 04 May 2022 01:44 +00:00: > It would be helpful if tests that are expected to fail could list both > the expected "correct" result (that is, the result if the test were to > succeed, which would cause the Xfail test to fail) and the expected > "wrong" output ... so that if the way a test goes wrong changes, that > too can be seen. > Fair request. FWIW, I don't know of any other test harness that supports this. > Yes, this can be done now by repeating the test twice, once as an > xfail and once as a regular test but with the "broken" result ... but > it might be easier to keep track of if it were all in one place. > How about if, for the time being, we use the "duplicate the test" approach, with F: lines or comments as needed? Once we have some experience with that, we may have a better idea of what test harness functionality to implement. > (No action on this expected pre-5.9-release, just musing.)