Test issues

Test issues

[Vin Shelton]

Greetings -

I'd like to bring up a few issues with the test scripts:

1. Since yesterday, the A04 test has been failing on my linux systems
at home (kubuntu) and at work (RHEL).  Here's the tail of
'ZTST_verbose=3 TESTNUM=A04 make test':

Running test: Using {fdvar}> syntax to open a new file descriptor
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.26465, output: /tmp/zsh.ztst.out.26465, error:
/tmp/zsh.ztst.terr.26465
make: [test] Error 127 (ignored)

Please let me know if you need more diagnostic input.

2. Peter, thanks for implementing the multibyte test.  There's a very
minor typo:

Index: Test/D07multibyte.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
retrieving revision 1.5
diff -u -r1.5 D07multibyte.ztst
--- Test/D07multibyte.ztst      10 Jul 2006 13:08:24 -0000      1.5
+++ Test/D07multibyte.ztst      11 Jul 2006 14:17:52 -0000
@@ -12,7 +12,7 @@
     fi
   done
   if [[ -z $mb_ok ]]; then
-    ZTST_unimplemented="no UTF-8 locale or multibyte mode is not implemnted"
+    ZTST_unimplemented="no UTF-8 locale or multibyte mode is not implemented"
   else
     print Testing multibyte with locale $LANG >&8
   fi

3. Is there a relliable way to get the number of test failures?  In
case #1 above, the output of 'make check' looks like this:

/usr/local/src/zsh-2006-07-11/Test/A03quoting.ztst: starting.
/usr/local/src/zsh-2006-07-11/Test/A03quoting.ztst: all tests successful.
/usr/local/src/zsh-2006-07-11/Test/A04redirect.ztst: starting.
/usr/local/src/zsh-2006-07-11/Test/A05execution.ztst: starting.
/usr/local/src/zsh-2006-07-11/Test/A05execution.ztst: all tests successful.

Note that there's no failure message printed for A04.  What I'm
currently doing is using 'grep -c' to count 'starting' and then
counting 'skipped' and 'successful'.  Currently, my math looks like
this: #failures = #starts - #successful - #skipped.  Is there a better
way?

Peter - I think the test suite is very helpful - thanks for creating it!

Thanks,
  Vin

-- 
Whoever you are, no matter how lonely,
the world offers itself to your imagination,
calls to you like the wild geese, harsh and exciting--
over and over announcing your place
in the family of things.			Mary Oliver

Re: Test issues

[Peter Stephenson]

"Vin Shelton" wrote:
> Greetings -
> 
> I'd like to bring up a few issues with the test scripts:
> 
> 1. Since yesterday, the A04 test has been failing on my linux systems
> at home (kubuntu) and at work (RHEL).  Here's the tail of
> 'ZTST_verbose=3 TESTNUM=A04 make test':
> 
> Running test: Using {fdvar}> syntax to open a new file descriptor
> ZTST_test: expecting status: 0
> Input: /tmp/zsh.ztst.in.26465, output: /tmp/zsh.ztst.out.26465, error:
> /tmp/zsh.ztst.terr.26465
> make: [test] Error 127 (ignored)

Thanks for spotting that... the feature you noticed that it was
failing silently must have been why I didn't.  I'm not sure why it
was doing that, but the problem must have made the entire test script
fall over somehow.  The bug it found was indeed parsing the syntax the
test was designed for.  I've extended the test to spot the failure
earlier.

I suppose the make file could check for missing "successful" or "failed"
messages.

Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.57
diff -u -r1.57 parse.c
--- Src/parse.c	10 Jul 2006 13:08:23 -0000	1.57
+++ Src/parse.c	11 Jul 2006 15:27:51 -0000
@@ -1603,7 +1603,7 @@
 
 		if (*ptr == Outbrace && ptr > tokstr + 1)
 		{
-		    if (itype_end(tokstr, IIDENT, 0) >= ptr - 1)
+		    if (itype_end(tokstr+1, IIDENT, 0) >= ptr - 1)
 		    {
 			char *toksave = tokstr;
 			char *idstring = dupstrpfx(tokstr+1, eptr-tokstr-1);
Index: Test/A04redirect.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A04redirect.ztst,v
retrieving revision 1.10
diff -u -r1.10 A04redirect.ztst
--- Test/A04redirect.ztst	23 Sep 2005 17:03:17 -0000	1.10
+++ Test/A04redirect.ztst	11 Jul 2006 15:27:51 -0000
@@ -240,10 +240,15 @@
 <input
 >input
 
+  myfd=
   exec {myfd}>logfile
-  print This is my logfile. >&$myfd
-  print Examining contents of logfile...
-  cat logfile
+  if [[ -z $myfd ]]; then
+    print "Ooops, failed to set myfd to a file descriptor." >&2
+  else
+    print This is my logfile. >&$myfd
+    print Examining contents of logfile...
+    cat logfile
+  fi
 0:Using {fdvar}> syntax to open a new file descriptor
 >Examining contents of logfile...
 >This is my logfile.
Index: Test/D07multibyte.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/D07multibyte.ztst,v
retrieving revision 1.5
diff -u -r1.5 D07multibyte.ztst
--- Test/D07multibyte.ztst	10 Jul 2006 13:08:24 -0000	1.5
+++ Test/D07multibyte.ztst	11 Jul 2006 15:27:52 -0000
@@ -12,7 +12,7 @@
     fi
   done
   if [[ -z $mb_ok ]]; then
-    ZTST_unimplemented="no UTF-8 locale or multibyte mode is not implemnted"
+    ZTST_unimplemented="no UTF-8 locale or multibyte mode is not implemented"
   else
     print Testing multibyte with locale $LANG >&8
   fi

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Test issues

[Bart Schaefer]

On Jul 11,  7:49am, Vin Shelton wrote:
} 
} Running test: Using {fdvar}> syntax to open a new file descriptor
} ZTST_test: expecting status: 0
} Input: /tmp/zsh.ztst.in.26465, output: /tmp/zsh.ztst.out.26465, error:
} /tmp/zsh.ztst.terr.26465
} make: [test] Error 127 (ignored)

The {myfd}> syntax has stopped working:

torch% exec {myfd}>logfile
zsh: command not found: {myfd}

(and zsh exits, because of the exec).

Re: Test issues

[Bart Schaefer]

On Jul 11,  4:29pm, Peter Stephenson wrote:
}
} +  myfd=
}    exec {myfd}>logfile
} -  print This is my logfile. >&$myfd
} -  print Examining contents of logfile...
} -  cat logfile
} +  if [[ -z $myfd ]]; then
} +    print "Ooops, failed to set myfd to a file descriptor." >&2

That's not going to help.  The problem is that if {mytfd} is not
recognized as a file descriptor, it's taken as a command, and the
"exec" causes the test script to replace itself with nothing (and
effectively exit).

The real solution is to put the entire thing in a subshell, I think.

Re: Test issues

[Peter Stephenson]

This puts tests for exec's inside subshells.

It also changes the test harness so that substitutions on variables
in output are performed after the test has run, making them rather
more useful.

That relies on the last byte of the output or error for each test being
a newline.  That happens to be true, but I wondered if there was any way
around the fact that even "$(<file)" strips the last newline and doesn't
give a way of finding out if there was one there (except for examining
the file itself).

Index: Test/A04redirect.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/A04redirect.ztst,v
retrieving revision 1.11
diff -u -r1.11 A04redirect.ztst
--- Test/A04redirect.ztst	11 Jul 2006 15:36:37 -0000	1.11
+++ Test/A04redirect.ztst	12 Jul 2006 11:45:35 -0000
@@ -82,18 +82,22 @@
 >b
 >c
 
-  exec 3>redir  &&  print hello >&3  &&  print goodbye >&3  && cat redir
+  #
+  # exec tests: perform these in subshells so if they fail the
+  # shell won't exit.
+  #
+  (exec 3>redir  &&  print hello >&3  &&  print goodbye >&3  && cat redir)
 0:'>&' redirection
 >hello
 >goodbye
 
-  exec 3<redir && read foo <&3 && print $foo && read foo <&3 && print $foo
+  (exec 3<redir && read foo <&3 && print $foo && read foo <&3 && print $foo)
 0:'<&' redirection
 >hello
 >goodbye
 
-  exec 3<&-
-  read foo <&-
+  (exec 3<&-
+  read foo <&-)
 1:'<&-' redirection
 
   print foo >&-
@@ -240,7 +244,7 @@
 <input
 >input
 
-  myfd=
+  (myfd=
   exec {myfd}>logfile
   if [[ -z $myfd ]]; then
     print "Ooops, failed to set myfd to a file descriptor." >&2
@@ -248,30 +252,36 @@
     print This is my logfile. >&$myfd
     print Examining contents of logfile...
     cat logfile
-  fi
+  fi)
 0:Using {fdvar}> syntax to open a new file descriptor
 >Examining contents of logfile...
 >This is my logfile.
 
-  setopt noclobber
-  exec {myfd}>logfile2
+  (setopt noclobber
+   exec {myfd}>logfile2
+   echo $myfd
+   exec {myfd}>logfile3) | read myfd
+  (( ! ${pipestatus[1]} ))
 1q:NO_CLOBBER prevents overwriting parameter with allocated fd
-?(eval):2: can't clobber parameter myfd containing file descriptor $myfd
+?(eval):4: can't clobber parameter myfd containing file descriptor $myfd
 
+  (exec {myfd}>logfile4
+  echo $myfd
   exec {myfd}>&-
-  print This message should disappear >&$myfd
+  print This message should disappear >&$myfd) | read myfd
+  (( ! ${pipestatus[1]} ))
 1q:Closing file descriptor using brace syntax
-?(eval):2: $myfd:$bad_fd_msg
+?(eval):4: $myfd:$bad_fd_msg
 
   typeset -r myfd
   echo This should not appear {myfd}>nologfile
 1:Error opening file descriptor using readonly variable
 ?(eval):2: can't allocate file descriptor to readonly parameter myfd
 
-  typeset +r myfd
+  (typeset +r myfd
   exec {myfd}>newlogfile
   typeset -r myfd
-  exec {myfd}>&-
+  exec {myfd}>&-)
 1:Error closing file descriptor using readonly variable
 ?(eval):4: can't close file descriptor from readonly parameter myfd
 
Index: Test/ztst.zsh
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/ztst.zsh,v
retrieving revision 1.23
diff -u -r1.23 ztst.zsh
--- Test/ztst.zsh	11 Oct 2005 16:48:06 -0000	1.23
+++ Test/ztst.zsh	12 Jul 2006 11:45:35 -0000
@@ -227,24 +227,25 @@
   ZTST_verbose 2 "ZTST_getredir: read redir for '$char':
 $ZTST_redir"
 
-case $char in
-  ('<') fn=$ZTST_in
-       ;;
-  ('>') fn=$ZTST_out
-       ;;
-  ('?') fn=$ZTST_err
-       ;;
-   (*)  ZTST_testfailed "bad redir operator: $char"
-       return 1
-       ;;
-esac
-if [[ $ZTST_flags = *q* ]]; then
-  print -r -- "${(e)ZTST_redir}" >>$fn
-else
-  print -r -- "$ZTST_redir" >>$fn
-fi
+  case $char in
+    ('<') fn=$ZTST_in
+    ;;
+    ('>') fn=$ZTST_out
+    ;;
+    ('?') fn=$ZTST_err
+    ;;
+    (*)  ZTST_testfailed "bad redir operator: $char"
+    return 1
+    ;;
+  esac
+  if [[ $ZTST_flags = *q* && $char = '<' ]]; then
+    # delay substituting output until variables are set
+    print -r -- "${(e)ZTST_redir}" >>$fn
+  else
+    print -r -- "$ZTST_redir" >>$fn
+  fi
 
-return 0
+  return 0
 }
 
 # Execute an indented chunk.  Redirections will already have
@@ -287,7 +288,7 @@
 }
     
 ZTST_test() {
-  local last match mbegin mend found
+  local last match mbegin mend found substlines
 
   while true; do
     rm -f $ZTST_in $ZTST_out $ZTST_err
@@ -375,6 +376,11 @@
 $(<$ZTST_terr)"
 
       # Now check output and error.
+      if [[ $ZTST_flags = *q* && -s $ZTST_out ]]; then
+	substlines="$(<$ZTST_out)"
+	rm -rf $ZTST_out
+	print -r -- "${(e)substlines}" >$ZTST_out
+      fi
       if [[ $ZTST_flags != *d* ]] && ! ZTST_diff -c $ZTST_out $ZTST_tout; then
 	ZTST_testfailed "output differs from expected as shown above for:
 $ZTST_code${$(<$ZTST_terr):+
@@ -382,6 +388,11 @@
 $(<$ZTST_terr)}"
 	return 1
       fi
+      if [[ $ZTST_flags = *q* && -s $ZTST_err ]]; then
+	substlines="$(<$ZTST_err)"
+	rm -rf $ZTST_err
+	print -r -- "${(e)substlines}" >$ZTST_err
+      fi
       if [[ $ZTST_flags != *D* ]] && ! ZTST_diff -c $ZTST_err $ZTST_terr; then
 	ZTST_testfailed "error output differs from expected as shown above for:
 $ZTST_code"

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070


To access the latest news from CSR copy this link into a web browser:  http://www.csr.com/email_sig.php

Re: Test issues

[Bart Schaefer]

On Jul 12, 12:48pm, Peter Stephenson wrote:
}
} That relies on the last byte of the output or error for each test being
} a newline.  That happens to be true, but I wondered if there was any way
} around the fact that even "$(<file)" strips the last newline and doesn't
} give a way of finding out if there was one there (except for examining
} the file itself).

It's a bit less efficient, but something like:

	"$(<file <<<$'\0')"

Of course that depends on multios, <<< and $'\0' all working correctly.