other testsuite problems

other testsuite problems

[Clint Adams]

In addition to the >&- test for which I have done no followup, I note
the following other problems in testsuiteland:

If $HOME does not exist or is not writable, many tests go awry.  Arguably, this is user error.

In some environments, I get this
../../Test/A01grammar.ztst: starting.
*** /tmp/zsh.ztst.err.22699	Mon Dec 24 21:24:33 2007
--- /tmp/zsh.ztst.terr.22699	Mon Dec 24 21:24:33 2007
***************
*** 1,2 ****
- 1) one     2) two     3) three   
  input> input> 
--- 1 ----
Test ../../Test/A01grammar.ztst failed: error output differs from expected as shown above for:
  (COLUMNS=80
  PS3="input> "
  select name in one two three; do
    print $name
  done)
Was testing: `select' loop
../../Test/A01grammar.ztst: test failed.

Finally, if D07multibyte.ztst is skipped due to a lack of UTF-8 locale, it seems as though that is
being registered as a failure. Perhaps a distinct exit code should be used for skips (and counting
such in runtests.zsh).

Re: other testsuite problems

[Clint Adams]

On Mon, Dec 24, 2007 at 05:23:25PM -0500, Clint Adams wrote:
> Finally, if D07multibyte.ztst is skipped due to a lack of UTF-8 locale, it seems as though that is
> being registered as a failure. Perhaps a distinct exit code should be used for skips (and counting
> such in runtests.zsh).

To that end,

Index: Test/runtests.zsh
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/runtests.zsh,v
retrieving revision 1.1
diff -u -r1.1 runtests.zsh
--- Test/runtests.zsh	18 Dec 2007 21:16:30 -0000	1.1
+++ Test/runtests.zsh	25 Dec 2007 01:35:00 -0000
@@ -7,10 +7,13 @@
 # protect from catastrophic failure of an individual test.
 # We could probably do that with subshells instead.
 
-integer success failure
+integer success failure skipped retval
 for file in "${(f)ZTST_testlist}"; do
   $ZTST_exe +Z -f $ZTST_srcdir/ztst.zsh $file
-  if (( $? )); then
+  retval=$?
+  if (( $retval == 2 )); then
+    (( skipped++ ))
+  elif (( $retval )); then
     (( failure++ ))
   else
     (( success++ ))
@@ -18,6 +21,7 @@
 done
 print "**************************************
 $success successful test script${${success:#1}:+s}, \
-$failure failure${${failure:#1}:+s}
+$failure failure${${failure:#1}:+s}, \
+$skipped skipped
 **************************************"
 return $(( failure ? 1 : 0 ))
Index: Test/ztst.zsh
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/ztst.zsh,v
retrieving revision 1.28
diff -u -r1.28 ztst.zsh
--- Test/ztst.zsh	18 Dec 2007 21:16:30 -0000	1.28
+++ Test/ztst.zsh	25 Dec 2007 01:35:00 -0000
@@ -467,6 +467,7 @@
 
 if [[ -n "$ZTST_unimplemented" ]]; then
   print "$ZTST_testname: skipped ($ZTST_unimplemented)"
+  ZTST_testfailed=2
 elif (( ! $ZTST_testfailed )); then
   print "$ZTST_testname: all tests successful."
 fi

Re: other testsuite problems

[Clint Adams]

On Mon, Dec 24, 2007 at 05:23:25PM -0500, Clint Adams wrote:
> Finally, if D07multibyte.ztst is skipped due to a lack of UTF-8 locale, it seems as though that is
> being registered as a failure. Perhaps a distinct exit code should be used for skips (and counting

This was completely off the mark; there is still a failure counted
despite it not being reported as failed (or I'm blind):
http://buildd.debian.org/fetch.cgi?pkg=zsh;ver=4.3.4-dev-6-3;arch=amd64;stamp=1198548543

Re: other testsuite problems

[Peter Stephenson]

On Mon, 24 Dec 2007 21:27:44 -0500
Clint Adams <clint@zsh.org> wrote:

> On Mon, Dec 24, 2007 at 05:23:25PM -0500, Clint Adams wrote:
> > Finally, if D07multibyte.ztst is skipped due to a lack of UTF-8 locale, it seems as though that is
> > being registered as a failure. Perhaps a distinct exit code should be used for skips (and counting
> 
> This was completely off the mark; there is still a failure counted
> despite it not being reported as failed (or I'm blind):
> http://buildd.debian.org/fetch.cgi?pkg=zsh;ver=4.3.4-dev-6-3;arch=amd64;stamp=1198548543

The fix is OK, but V01 is going off into a parallel universe; probably an
exec or similar.

../../Test/V01zmodload.ztst: starting.
../../Test/V02zregexparse.ztst: starting.

Try with more detail.

It might be better to shift the messages (apart from the hash printing,
obviously) to runtests.zsh, as well as making failures more obvious, to
fix this.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: other testsuite problems

[Clint Adams]

On Thu, Jan 03, 2008 at 06:17:01PM +0000, Peter Stephenson wrote:
> ../../Test/V01zmodload.ztst: starting.
> ../../Test/V02zregexparse.ztst: starting.
> 
> Try with more detail.

http://buildd.debian.org/fetch.cgi?&pkg=zsh&ver=4.3.4-dev-6-7&arch=s390&stamp=1199419143&file=log

Re: other testsuite problems

[Peter Stephenson]

On Thu, 3 Jan 2008 23:14:08 -0500
Clint Adams <clint@zsh.org> wrote:
> On Thu, Jan 03, 2008 at 06:17:01PM +0000, Peter Stephenson wrote:
> > ../../Test/V01zmodload.ztst: starting.
> > ../../Test/V02zregexparse.ztst: starting.
> > 
> > Try with more detail.
> 
> http://buildd.debian.org/fetch.cgi?&pkg=zsh&ver=4.3.4-dev-6-7&arch=s390&stamp=1199419143&file=log

So it's failed to load all the modules and crashed here:

 for m in $mods
 do
   zmodload -i $m || mods[(r)$m]=()
 done

You need to see which module is failing to load and where.  Outputting the module
with "print -u $ZTST_fd $mod" should help.  Then you can probably debug it on the
command line.

pws

Re: other testsuite problems

[Clint Adams]

On Fri, Jan 04, 2008 at 12:55:26PM +0000, Peter Stephenson wrote:
> You need to see which module is failing to load and where.  Outputting the module
> with "print -u $ZTST_fd $mod" should help.  Then you can probably debug it on the
> command line.

http://buildd.debian.org/fetch.cgi?&pkg=zsh&ver=4.3.4-dev-6-9&arch=alpha&stamp=1199472337&file=log

I could have sworn that this predates the switch to ncursesw, but I also
haven't had any trouble loading zsh/termcap by hand.

Re: other testsuite problems

[Peter Stephenson]

On Fri, 4 Jan 2008 14:05:45 -0500
Clint Adams <clint@zsh.org> wrote:
> On Fri, Jan 04, 2008 at 12:55:26PM +0000, Peter Stephenson wrote:
> > You need to see which module is failing to load and where.
> > Outputting the module with "print -u $ZTST_fd $mod" should help.
> > Then you can probably debug it on the command line.
> 
> http://buildd.debian.org/fetch.cgi?&pkg=zsh&ver=4.3.4-dev-6-9&arch=alpha&stamp=1199472337&file=log
> 
> I could have sworn that this predates the switch to ncursesw, but I also
> haven't had any trouble loading zsh/termcap by hand.

You might get further with strace, I suppose.  Or you might get the same
effect loading all the modules one by one.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: other testsuite problems

[Clint Adams]

On Sat, Jan 05, 2008 at 12:25:39PM +0000, Peter Stephenson wrote:
> You might get further with strace, I suppose.  Or you might get the same
> effect loading all the modules one by one.

I am having no success reproducing the `make check' failure under any
interactive circumstances.