There seems to be a problem with the test harness on Solaris 11.4

There seems to be a problem with the test harness on Solaris 11.4

[Clinton Bunch]

Every test failed with an error similar to the one down below.


./A01grammar.ztst: starting.
diff: illegal option -- a
usage: diff [-bitw] [-c | -e | -f | -h | -n | -u] file1 file2
        diff [-bitw] [-C number | -U number] file1 file2
        diff [-bitw] [-D string] file1 file2
        diff [-bitw] [-c | -e | -f | -h | -n | -u] [-l] [-r] [-s] [-S 
name] directory1 directory2

Test ./A01grammar.ztst failed: output differs from expected as shown 
above for:
   # Test skipping early to ensure we run the remainder...
   if [[ -n $ZTST_test_skip ]]; then
     ZTST_skip="Test system verification for skipping"
   else
     print "This is standard output"
     print "This is standard error" >&2
     false
   fi
Error output:
This is standard error
Was testing: Test skipping if ZTST_test_skip is set
./A01grammar.ztst: test failed.

Re: There seems to be a problem with the test harness on Solaris 11.4

[Bart Schaefer]

On Sun, Nov 6, 2022 at 8:56 AM Clinton Bunch <cdb_zsh@zentaur.org> wrote:
>
> ./A01grammar.ztst: starting.
> diff: illegal option -- a

For GNU diff, -a means to forcibly compare files even if they seem to
contain non-text bytes.  What's the corresponding Solaris option (and
rhetorically, why the heck did they drop -a)?

Re: There seems to be a problem with the test harness on Solaris 11.4

[Clinton Bunch]

On 11/6/2022 12:12 PM, Bart Schaefer wrote:
> On Sun, Nov 6, 2022 at 8:56 AM Clinton Bunch <cdb_zsh@zentaur.org> wrote:
>> ./A01grammar.ztst: starting.
>> diff: illegal option -- a
> For GNU diff, -a means to forcibly compare files even if they seem to
> contain non-text bytes.  What's the corresponding Solaris option (and
> rhetorically, why the heck did they drop -a)?
>
A look at the man page for diff doesn't show anything similar. It 
doesn't look like it wasn't in the diff for Solaris 10.  I even check 
Solaris 7 and there wasn't an equivalent option that I could find.

Re: There seems to be a problem with the test harness on Solaris 11.4

[Clinton Bunch]

On 11/6/2022 12:28 PM, Clinton Bunch wrote:
> On 11/6/2022 12:12 PM, Bart Schaefer wrote:
>> On Sun, Nov 6, 2022 at 8:56 AM Clinton Bunch <cdb_zsh@zentaur.org> 
>> wrote:
>>> ./A01grammar.ztst: starting.
>>> diff: illegal option -- a
>> For GNU diff, -a means to forcibly compare files even if they seem to
>> contain non-text bytes.  What's the corresponding Solaris option (and
>> rhetorically, why the heck did they drop -a)?
>>
> A look at the man page for diff doesn't show anything similar. It 
> doesn't look like it wasn't in the diff for Solaris 10.  I even check 
> Solaris 7 and there wasn't an equivalent option that I could find.
>
>
A look at the HP-UX man page shows a similar lack.  But it looks like 
Solaris 11.4 at least also has GNU diff as gdiff.

Re: There seems to be a problem with the test harness on Solaris 11.4

[Bart Schaefer]

On Sun, Nov 6, 2022 at 10:35 AM Clinton Bunch <cdb_zsh@zentaur.org> wrote:
>
> Solaris 11.4 at least also has GNU diff as gdiff.

Use of "diff -a" hasn't changed since at least 2017.  Is it possible
that there's been a default (or your own) PATH change causing the
POSIX diff to be found in preference?

Re: There seems to be a problem with the test harness on Solaris 11.4

[Clinton Bunch]

On 11/6/2022 1:09 PM, Bart Schaefer wrote:
> On Sun, Nov 6, 2022 at 10:35 AM Clinton Bunch <cdb_zsh@zentaur.org> wrote:
>> Solaris 11.4 at least also has GNU diff as gdiff.
> Use of "diff -a" hasn't changed since at least 2017.  Is it possible
> that there's been a default (or your own) PATH change causing the
> POSIX diff to be found in preference?
>
I haven't changed the default path on this system yet.  To get GNU diff 
as diff you'd have to add /usr/gnu/bin to the path, and I doubt that's 
been in the standard path.  I'm not positive it wasn't pulled in as a 
prereq when I installed autoconf from the standard package repository.

I think it's possible that only a few people have run the tests on 
Solaris in the last 5 years and either had /usr/gnu/bin in their path 
(not exactly unexpected from someone who'd try to build their own shell) 
or they just didn't report the failure.

Especially since it ships with a version of zsh (5.3.1), a lot of people 
wouldn't feel the need to build their own.

Re: There seems to be a problem with the test harness on Solaris 11.4

[Clinton Bunch]

On 11/6/2022 1:32 PM, Clinton Bunch wrote:
> On 11/6/2022 1:09 PM, Bart Schaefer wrote:
>> On Sun, Nov 6, 2022 at 10:35 AM Clinton Bunch <cdb_zsh@zentaur.org> 
>> wrote:
>>> Solaris 11.4 at least also has GNU diff as gdiff.
>> Use of "diff -a" hasn't changed since at least 2017.  Is it possible
>> that there's been a default (or your own) PATH change causing the
>> POSIX diff to be found in preference?
>>
>
>
>
I'm working on a patch to check if diff supports -a, check if gdiff is 
in the path.  If both of those fail, do we want to run without -a or 
abort and kindly suggest they install GNU diff?

[PATCH] diff for test harness (was: There seems to be a problem with the test harness on Solaris 11.4)

[Clinton Bunch]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

On 11/7/2022 1:39 PM, Clinton Bunch wrote:
> On 11/6/2022 1:32 PM, Clinton Bunch wrote:
>> On 11/6/2022 1:09 PM, Bart Schaefer wrote:
>>> On Sun, Nov 6, 2022 at 10:35 AM Clinton Bunch <cdb_zsh@zentaur.org> 
>>> wrote:
>>>> Solaris 11.4 at least also has GNU diff as gdiff.
>>> Use of "diff -a" hasn't changed since at least 2017.  Is it possible
>>> that there's been a default (or your own) PATH change causing the
>>> POSIX diff to be found in preference?
>>>
>>
>>
>>
> I'm working on a patch to check if diff supports -a, check if gdiff is 
> in the path.  If both of those fail, do we want to run without -a or 
> abort and kindly suggest they install GNU diff?
>
Here's a patch that has configure check for an appropriate diff in the 
path and if it fails to find one runs diff without -a in the test harness


[-- Attachment #2: zsh-diff-a.patch --]
[-- Type: text/plain, Size: 2064 bytes --]

diff --git a/Config/defs.mk.in b/Config/defs.mk.in
index 2bc17482a..c979307f7 100644
--- a/Config/defs.mk.in
+++ b/Config/defs.mk.in
@@ -75,6 +75,7 @@ IMPOPT          = @IMPOPT@
 
 # utilities
 AWK             = @AWK@
+DIFF            = @DIFF@
 ANSI2KNR        = @ANSI2KNR@
 YODL            = @YODL@ @YODL_OPTIONS@
 YODL2TXT        = @YODL@2txt
diff --git a/Test/Makefile.in b/Test/Makefile.in
index 09f37bf53..636746a45 100644
--- a/Test/Makefile.in
+++ b/Test/Makefile.in
@@ -53,6 +53,7 @@ check test:
 	if ZTST_testlist="`for f in $(sdir)/$(TESTNUM)*.ztst; \
            do echo $$f; done`" \
 	 ZTST_srcdir="$(sdir)" \
+	 ZTST_diff_pg="$(DIFF)" \
 	 ZTST_exe=$(dir_top)/Src/zsh@EXEEXT@ \
 	 $(dir_top)/Src/zsh@EXEEXT@ +Z -f $(sdir)/$$ZTST_handler; then \
 	 stat=0; \
diff --git a/Test/ztst.zsh b/Test/ztst.zsh
index ea1b016d5..96a138483 100755
--- a/Test/ztst.zsh
+++ b/Test/ztst.zsh
@@ -382,7 +382,7 @@ ZTST_diff() {
       diff_ret=1
     fi
   else
-    diff_out=$(diff -a "$@")
+    diff_out=$(${=ZTST_diff_pg}  "$@")
     diff_ret="$?"
     if [[ "$diff_ret" != "0" ]]; then
       print -r -- "$diff_out"
diff --git a/configure.ac b/configure.ac
index 074141d38..7e9426148 100644
--- a/configure.ac
+++ b/configure.ac
@@ -611,6 +611,19 @@ AC_PROG_AWK                 dnl Check for mawk,gawk,nawk, then awk.
 AC_PROG_LN                  dnl Check for working ln, for "make install"
 AC_PROG_LN_S                dnl Use ln -s/ln/cp for "make install.runhelp"
 AC_PROG_EGREP               dnl sets $EGREP to grep -E or egrep
+
+dnl check for diff that supports -a
+AC_CACHE_CHECK([for diff that supports -a],[ac_cv_path_DIFF],
+    [AC_PATH_PROGS_FEATURE_CHECK([DIFF],[diff gdiff],
+       [[
+	    echo "abcdef" > conftest.$$
+	    echo "abcdef" > conftest2.$$
+	    $ac_path_DIFF -a conftest.$$ conftest2.$$ > /dev/null 2>&1
+	    test "x$?" = x0 \
+            && ac_cv_path_DIFF="$ac_path_DIFF -a" ac_path_DIFF_found=:]],
+	[ac_cv_path_DIFF="diff"])])
+AC_SUBST([DIFF],[$ac_cv_path_DIFF])
+
 AC_CHECK_PROGS([YODL], [yodl], [: yodl])
 
 YODL_OPTIONS=''