Output of 'make check'

Output of 'make check'

[Andrej Borsenkow]

I do not particular like it. It outputs thousands of "No differences
encountered", that will scroll away any useful output. So, if any test
in between fails, there is no chance to see it unfortunately (there
seems to be no log by default). I just noticed, that one more test
failed - and it was only by accident.

I suggest, that "No differences encountered" should not be output (I do
not see test numbers anyway, so it pretty useless)

some summary about failed test is output, so that I can at least see I
had failed tests. If summary coould indicate failed test number - even
better

-andrej

Have a nice DOS!
B >>

test failuer case

[Andrej Borsenkow]

Running with current CVS (including last patch from Peter) gives me:

*** /tmp/zsh.ztst.out.25823     Thu Jun  1 13:22:33 2000
--- /tmp/zsh.ztst.tout.25823    Thu Jun  1 13:22:34 2000
***************
*** 1 ****
! line: {tst -x }{}
--- 1,2 ----
! line: {tst -}{}
! MESSAGE:{arg}
Test /tools/src/zsh/Test/53completion.ztst failed: output differs from
expected as shown above for:
 comptesteval 'compdef _tst tst; _tst () { _arguments "-x" ":arg:" }'
 comptest $'tst -\t'
Was testing: _arguments
/tools/src/zsh/Test/53completion.ztst: test failed.

I cannot say, if it failed before, because output may have been scrolled
away.

This is in addition to cursor positioning problem (this consistently
fails here).

-andrej

Have a nice DOS!
B >>

Re: Output of 'make check'

[Peter Stephenson]

> I do not particular like it. It outputs thousands of "No differences
> encountered", that will scroll away any useful output. So, if any test
> in between fails, there is no chance to see it unfortunately (there
> seems to be no log by default). I just noticed, that one more test
> failed - and it was only by accident.

That's coming from your version of diff, not the test harness.  If you can
find a portable way of suppressing it without removing the useful output, I
will add it --- it may simply be a case of 2>/dev/null.

I'm getting the _arguments test failure you report, too, but _arguments is
not something I'd like to see changed at this stage, since a lot of the
completion system depends on quite small details of how it works.  We can
leave the test out for the time being.

We could create a separate set of tests known to fail, not run by default
but designed to embarrass the developers.  I'm not sure how much I like the
sound of that.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

RE: Output of 'make check'

[Andrej Borsenkow]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

>
> > I do not particular like it. It outputs thousands of "No differences
> > encountered", that will scroll away any useful output. So,
> if any test
> > in between fails, there is no chance to see it unfortunately (there
> > seems to be no log by default). I just noticed, that one more test
> > failed - and it was only by accident.
>
> That's coming from your version of diff, not the test
> harness.  If you can
> find a portable way of suppressing it without removing the
> useful output, I
> will add it --- it may simply be a case of 2>/dev/null.
>

What about attached patch? (2>/dev/null does not work, diff outputs on
stdout).

Can we rely on the fact, that $? is set to exit code of command
substitution?

-andrej

[-- Attachment #2: ztst.diff --]
[-- Type: application/octet-stream, Size: 1043 bytes --]

--- Test/ztst.zsh.org	Wed May  3 13:24:47 2000
+++ Test/ztst.zsh	Thu Jun  1 16:54:31 2000
@@ -219,6 +219,19 @@
   done
 }
 
+# diff wrapper
+ZTST_diff() {
+  local diff_out diff_ret
+
+  diff_out=$(diff "$@")
+  diff_ret="$?"
+  if [[ "$diff_ret" != "0" ]]; then
+    echo "$diff_out"
+  fi
+
+  return "$diff_ret"
+}
+    
 ZTST_test() {
   local last match mbegin mend found
 
@@ -297,13 +310,13 @@
 $(<$ZTST_terr)"
 
       # Now check output and error.
-      if [[ $ZTST_flags != *d* ]] && ! diff -c $ZTST_out $ZTST_tout; then
+      if [[ $ZTST_flags != *d* ]] && ! ZTST_diff -c $ZTST_out $ZTST_tout; then
 	ZTST_testfailed "output differs from expected as shown above for:
 $ZTST_code${$(<$ZTST_terr):+
 Error output:
 $(<$ZTST_terr)}"
       fi
-      if [[ $ZTST_flags != *D* ]] && ! diff -c $ZTST_err $ZTST_terr; then
+      if [[ $ZTST_flags != *D* ]] && ! ZTST_diff -c $ZTST_err $ZTST_terr; then
 	ZTST_testfailed "error output differs from expected as shown above for:
 $ZTST_code"
       fi

Re: test failuer case

[Bart Schaefer]

On Jun 1,  1:29pm, Andrej Borsenkow wrote:
} Subject: test failuer case
}
} Running with current CVS (including last patch from Peter) gives me:
} 
} *** /tmp/zsh.ztst.out.25823     Thu Jun  1 13:22:33 2000
} --- /tmp/zsh.ztst.tout.25823    Thu Jun  1 13:22:34 2000
} ***************
} *** 1 ****
} ! line: {tst -x }{}
} --- 1,2 ----
} ! line: {tst -}{}
} ! MESSAGE:{arg}

This is the test that we changed right before 3.1.7 so that "make check"
would be clean (11647), when in fact it was telling us about a real bug
which Sven has subsequently fixed (11678).  We should put the test back
the way it was intended to be.

Index: Test/53comptest.zsh
======================================================================
@@ -129,7 +129,8 @@
  comptesteval 'compdef _tst tst; _tst () { _arguments "-x" ":arg:" }'
  comptest $'tst -\t'
 0:_arguments
->line: {tst -x }{}
+>line: {tst -}{}
+>MESSAGE:{arg}
 
  comptesteval 'compdef _tst tst; _tst () { _arguments "-x:arg:" }'
  comptest $'tst -x\t'


-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Output of 'make check'

[Bart Schaefer]

On Jun 1,  5:03pm, Andrej Borsenkow wrote:
} Subject: RE: Output of 'make check'
}
} Can we rely on the fact, that $? is set to exit code of command
} substitution?

If we can't, then that's a different test that ought to fail.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Output of 'make check'

[Peter Stephenson]

> What about attached patch? (2>/dev/null does not work, diff outputs on
> stdout).

Looks about the best way of doing it --- we don't want to start mucking
with options to different versions of diff.  I won't apply it until after
the next trial release, just in case it tickles something.

> Can we rely on the fact, that $? is set to exit code of command
> substitution?

Yes, that works in the case that the command substitution is the only thing
on the right hand side of an assignment --- normally you don't get it's
status, of course.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070