PATCH: count test failures

PATCH: count test failures

[Peter Stephenson]

This adds an intermediate script in the Test directory to report test
scripts that succeeded and failed.  This is for lazy people who can't be
bothered to scroll the terminal.  We already make far more demands of
the shell in ztst.zsh than in runtests.zsh, so I can't see an obvious
problem.  But I'm sure you'll let me know.

Index: Test/.distfiles
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/.distfiles,v
retrieving revision 1.19
diff -u -r1.19 .distfiles
--- Test/.distfiles	1 Aug 2007 21:38:35 -0000	1.19
+++ Test/.distfiles	18 Dec 2007 20:48:10 -0000
@@ -1,15 +1,41 @@
 DISTFILES_SRC='
+.cvsignore
 .distfiles
-.cvsignore           B01cd.ztst           D03procsubst.ztst    Y02compmatch.ztst
-C01arith.ztst        D04parameter.ztst    Y03arguments.ztst
-A01grammar.ztst      C02cond.ztst         D05array.ztst        comptest
-A02alias.ztst        C03traps.ztst        E02xtrace.ztst
-A03quoting.ztst      C04funcdef.ztst      Makefile.in          ztst.zsh
-A04redirect.ztst     D01prompt.ztst       V02zregexparse.ztst
-A05execution.ztst    D02glob.ztst         Y01completion.ztst
-D06subscript.ztst    D07multibyte.ztst    D08cmdsubst.ztst
-V01zmodload.ztst     E01options.ztst
-B02typeset.ztst      B03print.ztst        A06assign.ztst       B04read.ztst
-V03mathfunc.ztst     V04features.ztst     V05styles.ztst
+A01grammar.ztst
+A02alias.ztst
+A03quoting.ztst
+A04redirect.ztst
+A05execution.ztst
+A06assign.ztst
+B01cd.ztst
+B02typeset.ztst
+B03print.ztst
+B04read.ztst
+C01arith.ztst
+C02cond.ztst
+C03traps.ztst
+C04funcdef.ztst
+D01prompt.ztst
+D02glob.ztst
+D03procsubst.ztst
+D04parameter.ztst
+D05array.ztst
+D06subscript.ztst
+D07multibyte.ztst
+D08cmdsubst.ztst
+E01options.ztst
+E02xtrace.ztst
+Makefile.in
 README
+V01zmodload.ztst
+V02zregexparse.ztst
+V03mathfunc.ztst
+V04features.ztst
+V05styles.ztst
+Y01completion.ztst
+Y02compmatch.ztst
+Y03arguments.ztst
+comptest
+runtests.zsh
+ztst.zsh
 '
Index: Test/Makefile.in
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/Makefile.in,v
retrieving revision 1.8
diff -u -r1.8 Makefile.in
--- Test/Makefile.in	26 Jul 2004 13:18:14 -0000	1.8
+++ Test/Makefile.in	18 Dec 2007 20:48:10 -0000
@@ -45,10 +45,11 @@
 	  cd $(dir_top) && DESTDIR= \
 	  $(MAKE) MODDIR=`pwd`/$(subdir)/Modules install.modules > /dev/null; \
 	fi
-	-for f in $(sdir)/$(TESTNUM)*.ztst; do \
-	  ZTST_exe=$(dir_top)/Src/zsh \
-	    $(dir_top)/Src/zsh +Z -f $(sdir)/ztst.zsh $$f; \
-	done
+	-ZTST_testlist="`for f in $(sdir)/$(TESTNUM)*.ztst; \
+           do echo $$f; done`" \
+	 ZTST_srcdir="$(sdir)" \
+	 ZTST_exe=$(dir_top)/Src/zsh \
+	 $(dir_top)/Src/zsh +Z -f $(sdir)/runtests.zsh
 	rm -rf Modules .zcompdump
 
 # ========== DEPENDENCIES FOR CLEANUP ==========
Index: Test/runtests.zsh
===================================================================
RCS file: Test/runtests.zsh
diff -N Test/runtests.zsh
--- /dev/null	1 Jan 1970 00:00:00 -0000
+++ Test/runtests.zsh	18 Dec 2007 20:48:10 -0000
@@ -0,0 +1,22 @@
+#!/bin/zsh -f
+
+emulate zsh
+
+# Run all specified tests, keeping count of which succeeded.
+# The reason for this extra layer above the test script is to
+# protect from catastrophic failure of an individual test.
+# We could probably do that with subshells instead.
+
+integer success failure
+for file in "${(f)ZTST_testlist}"; do
+  $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh $file
+  if (( $? )); then
+    (( failure++ ))
+  else
+    (( success++ ))
+  fi
+done
+print "**************************************
+$success successful test script${${success:#1}:+s}, \
+$failure failure${${failure:#1}:+s}
+**************************************"
Index: Test/ztst.zsh
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/ztst.zsh,v
retrieving revision 1.27
diff -u -r1.27 ztst.zsh
--- Test/ztst.zsh	29 May 2007 14:50:29 -0000	1.27
+++ Test/ztst.zsh	18 Dec 2007 20:48:10 -0000
@@ -1,4 +1,4 @@
-#!/usr/local/bin/zsh -f
+#!/bin/zsh -f
 # The line above is just for convenience.  Normally tests will be run using
 # a specified version of zsh.  With dynamic loading, any required libraries
 # must already have been installed in that case.


-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: PATCH: count test failures

[Ismail Dönmez]

Tuesday 18 December 2007 22:52:14 tarihinde şunları yazmıştınız:
> This adds an intermediate script in the Test directory to report test
> scripts that succeeded and failed.  This is for lazy people who can't be
> bothered to scroll the terminal.  We already make far more demands of
> the shell in ztst.zsh than in runtests.zsh, so I can't see an obvious
> problem.  But I'm sure you'll let me know.

Hmm how about exiting with failure at the end if a test fails?



-- 
Never learn by your mistakes, if you do you may never dare to try again.

Re: PATCH: count test failures

[Peter Stephenson]

On Tue, 18 Dec 2007 22:56:22 +0200
Ismail Dönmez <ismail@pardus.org.tr> wrote:
> Hmm how about exiting with failure at the end if a test fails?

Makes sense; I'll tweak the Makefile and the script.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/