Y01 Test Failure

Y01 Test Failure

[Vin Shelton]

[-- Attachment #1: Type: text/plain, Size: 1026 bytes --]

Using the latest sources, I'm seeing:

../../../src/zsh-2021-03-18/Test/Y01completion.ztst: starting.
--- /tmp/zsh.ztst.1124192/ztst.out 2021-03-18 11:16:21.697760953 -0400
+++ /tmp/zsh.ztst.1124192/ztst.tout 2021-03-18 11:16:21.794427968 -0400
@@ -1,4 +1,3 @@
-NO:{<INSERT>menu</INSERT>}
 NO:{a}
 NO:{b}
 NO:{c}
@@ -8,6 +7,7 @@
 NO:{g}
 NO:{h}
 NO:{i}
+NO:{<INSERT>menu</INSERT>}
 NO:{j}
 NO:{k}
 NO:{l}
Test ../../../src/zsh-2021-03-18/Test/Y01completion.ztst failed: output
differs from expected as shown above for:
  comptesteval '_tst() { local disp=( {a..z} ); compadd -ld disp $disp[@];
comppostfuncs=( _pst ) }'
  comptesteval '_pst() { local disp=( "<INSERT>$compstate[insert]</INSERT>"
); compadd -Qld disp $disp }'
  comptesteval "zstyle ':completion:*' menu select=long-list"
  comptest $'tst \C-d'
Was testing:  menu select=long-list starts menu selection for list widgets
../../../src/zsh-2021-03-18/Test/Y01completion.ztst: test failed.
../../../src/zsh-2021-03-18/Test/Y02compmatch.ztst: starting.

HTH,
  Vin

[-- Attachment #2: Type: text/html, Size: 1716 bytes --]

Re: Y01 Test Failure

[Bart Schaefer]

On Thu, Mar 18, 2021 at 8:21 AM Vin Shelton <acs@alumni.princeton.edu> wrote:
>
> Using the latest sources, I'm seeing:

I have exactly the same failure on Ubuntu, test passes on MacOS.  Note
comptesteval does
stty 38400 columns 80 rows 24 tabs -icanon -iexten
and the test listing is 26 elements, so it would not fit on one
"screen" in a single column.

I don't know why that would behave differently on different OS.
Difference in terminal definition for "vt100"?

Re: Y01 Test Failure

[Bart Schaefer]

On Thu, Mar 18, 2021 at 8:21 AM Vin Shelton <acs@alumni.princeton.edu> wrote:
>
> @@ -8,6 +7,7 @@
>  NO:{g}
>  NO:{h}
>  NO:{i}
> +NO:{<INSERT>menu</INSERT>}
>  NO:{j}
>  NO:{k}
>  NO:{l}

If I change comptesteval to stty "rows 48" then this changes to:

@@ -8,6 +7,7 @@
 NO:{g}
 NO:{h}
 NO:{i}
+NO:{<INSERT></INSERT>}
 NO:{j}
 NO:{k}
 NO:{l}

So it does have something to do with tty size/definition.  Also looks
like the order in which the pseudo-tty driver is returning control
sequences may be different.

Re: Y01 Test Failure

[Jun T]

I *guess* the difference is in how the list is sorted.
On macOS '<INSERT>' comes before 'a', while on Linux it is between 'i' and 'j'.
I don't know why '<' is not used for sorting on Linux, but if <INSERT> is replaced
by <BBB>, for example, it is listed between 'b' and 'c' on Linux.

It would be sufficient to replace <INSERT> by any string whose sort order is more
stable across different OSs.

Re: Y01 Test Failure

[Mikael Magnusson]

On 3/19/21, Jun T <takimoto-j@kba.biglobe.ne.jp> wrote:
> I *guess* the difference is in how the list is sorted.
> On macOS '<INSERT>' comes before 'a', while on Linux it is between 'i' and
> 'j'.
> I don't know why '<' is not used for sorting on Linux, but if <INSERT> is
> replaced
> by <BBB>, for example, it is listed between 'b' and 'c' on Linux.
>
> It would be sufficient to replace <INSERT> by any string whose sort order is
> more
> stable across different OSs.

Is this happening even with LC_COLLATE=C, or did we not bother setting
that for this specific test?

-- 
Mikael Magnusson

Re: Y01 Test Failure

[Jun T]

> 2021/03/19 17:27, Mikael Magnusson <mikachu@gmail.com> wrote:
> 
> Is this happening even with LC_COLLATE=C, or did we not bother setting
> that for this specific test?

LC_ALL is set to en_US.UTF-8 at the start of Y01completion.ztst.

I've been thinking that comparison of all-ASCII strings is the same
in C and UTF-8 locales. But it turned out that strcoll() behaves quite
*strangely* under en_US.UTf-8 on Linux.

If I run the following C-code:

#include <stdio.h>
#include <string.h>
#include <locale.h>

int main() {
    char* s[] = { "h", "i", "j" };
    setlocale(LC_COLLATE, "");
    for(int i=0; i<3; ++i) {
        printf("'%s'  - '<INSERT>' = %d\n", s[i], strcoll(s[i], "<INSERT>"));
    }
    return 0;
}

% export LC_COLLATE=C          
% ./a.out
'h'  - '<INSERT>' = 44
'i'  - '<INSERT>' = 45
'j'  - '<INSERT>' = 46
% export LC_COLLATE=en_US.UTF-8
% ./a.out
'h'  - '<INSERT>' = -11
'i'  - '<INSERT>' = -1
'j'  - '<INSERT>' = 1
% export LC_COLLATE=ja_JP.UTF-8
% ./a.out
'h'  - '<INSERT>' = 44
'i'  - '<INSERT>' = 45
'j'  - '<INSERT>' = 46

I can't understand the behavior under en_US.UTF-8 locale.

Re: Y01 Test Failure

[dana]

Sorry i missed this, i was also using macOS when i tried it.

I only looked for a minute, but could we just do this? It seems to fix the
problem for me on Ubuntu, at least.

(Or, if it's useful in general, we could make comptest set LANG instead of
LC_ALL, and then force LC_COLLATE to C. But again i didn't check to see if
that would break something else)

dana


diff --git a/Test/Y01completion.ztst b/Test/Y01completion.ztst
index f6474c4a1..571f3cf16 100644
--- a/Test/Y01completion.ztst
+++ b/Test/Y01completion.ztst
@@ -240,7 +240,10 @@ F:regression test workers/31611
   comptesteval '_tst() { local disp=( {a..z} ); compadd -ld disp $disp[@]; comppostfuncs=( _pst ) }'
   comptesteval '_pst() { local disp=( "<INSERT>$compstate[insert]</INSERT>" ); compadd -Qld disp $disp }'
   comptesteval "zstyle ':completion:*' menu select=long-list"
+  # This test is sensitive to sorting differences across platforms
+  comptesteval 'export LC_ALL=C'
   comptest $'tst \C-d'
+  comptesteval "export LC_ALL=${(q)ZSH_TEST_LANG}"
 0: menu select=long-list starts menu selection for list widgets
 >NO:{<INSERT>menu</INSERT>}
 >NO:{a}

Re: Y01 Test Failure

[Vin Shelton]

[-- Attachment #1: Type: text/plain, Size: 1276 bytes --]

WFM here on Arch.

  - Vin

On Sun, Mar 21, 2021 at 10:10 PM dana <dana@dana.is> wrote:

> Sorry i missed this, i was also using macOS when i tried it.
>
> I only looked for a minute, but could we just do this? It seems to fix the
> problem for me on Ubuntu, at least.
>
> (Or, if it's useful in general, we could make comptest set LANG instead of
> LC_ALL, and then force LC_COLLATE to C. But again i didn't check to see if
> that would break something else)
>
> dana
>
>
> diff --git a/Test/Y01completion.ztst b/Test/Y01completion.ztst
> index f6474c4a1..571f3cf16 100644
> --- a/Test/Y01completion.ztst
> +++ b/Test/Y01completion.ztst
> @@ -240,7 +240,10 @@ F:regression test workers/31611
>    comptesteval '_tst() { local disp=( {a..z} ); compadd -ld disp
> $disp[@]; comppostfuncs=( _pst ) }'
>    comptesteval '_pst() { local disp=(
> "<INSERT>$compstate[insert]</INSERT>" ); compadd -Qld disp $disp }'
>    comptesteval "zstyle ':completion:*' menu select=long-list"
> +  # This test is sensitive to sorting differences across platforms
> +  comptesteval 'export LC_ALL=C'
>    comptest $'tst \C-d'
> +  comptesteval "export LC_ALL=${(q)ZSH_TEST_LANG}"
>  0: menu select=long-list starts menu selection for list widgets
>  >NO:{<INSERT>menu</INSERT>}
>  >NO:{a}
>
>
>

[-- Attachment #2: Type: text/html, Size: 2038 bytes --]

Re: Y01 Test Failure

[dana]

On 22 Mar 2021, at 06:06, Vin Shelton <acs@alumni.princeton.edu> wrote:
> WFM here on Arch.

Thanks. I committed it like that, we can revisit later if we want to make it
fancier

dana

Y02 test failure

[Marlon Richert]

On macOS:

% make TESTNUM=Y01 check
if test -n "gcc"; then \
  cd .. && DESTDIR= \
  /Applications/Xcode.app/Contents/Developer/usr/bin/make
MODDIR=`pwd`/Test/Modules install.modules > /dev/null; \
fi
if test -z "$ZTST_handler"; then \
  ZTST_handler=runtests.zsh; \
fi; \
if ZTST_testlist="`for f in ./Y01*.ztst; \
           do echo $f; done`" \
ZTST_srcdir="." \
ZTST_exe=../Src/zsh \
../Src/zsh +Z -f ./$ZTST_handler; then \
stat=0; \
else \
stat=1; \
fi; \
sleep 1; \
rm -rf Modules .zcompdump; \
exit $stat
./Y01completion.ztst: starting.
--- /tmp/zsh.ztst.176/ztst.out 2021-11-12 21:19:31.000000000 +0200
+++ /tmp/zsh.ztst.176/ztst.tout 2021-11-12 21:19:31.000000000 +0200
@@ -1 +1 @@
-line: {ztst.zsh }{}
+line: {ztst.zsh}{}
Test ./Y01completion.ztst failed: output differs from expected as
shown above for:
  comptesteval "path=( $ZTST_srcdir:A )"
  comptest $'zt\t'
Was testing: command
./Y01completion.ztst: test failed.
**************************************
0 successful test scripts, 1 failure, 0 skipped
**************************************
make: *** [check] Error 1

Re: Y02 test failure

[Bart Schaefer]

Your subject says Y02 but your text showed Y01.  In any case all tests
pass for me on both Ubuntu and MacOS.

My first suspicion is always a "build tree is not source tree"
problem, but from the diff I can't see how that would be it, in this
case.

Re: Y02 test failure

[Marlon Richert]

On Fri, Nov 12, 2021 at 9:57 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> Your subject says Y02 but your text showed Y01.

Whoops! It appears workers 49530 got stuck in my head.

> In any case all tests
> pass for me on both Ubuntu and MacOS.
> My first suspicion is always a "build tree is not source tree"
> problem, but from the diff I can't see how that would be it, in this
> case.

I tried it again after doing `make realclean`, etc., from both the
repo root and the Test dir. Still fails.

Re: Y02 test failure

[Bart Schaefer]

On Fri, Nov 12, 2021 at 1:28 PM Marlon Richert <marlon.richert@gmail.com> wrote:
>
> I tried it again after doing `make realclean`, etc.

Dunno what else to tell you; have you tried increasing ZTST_verbose ?

zsh-5.8-542-gc7f4634

  524  make realclean
  526  ./Util/preconfig
  527  ./configure
  528  make
  529  make check TESTNUM=Y01

cd Test ; /Applications/Xcode.app/Contents/Developer/usr/bin/make check
if test -n "gcc"; then \
      cd .. && DESTDIR= \
      /Applications/Xcode.app/Contents/Developer/usr/bin/make
MODDIR=`pwd`/Test/Modules install.modules > /dev/null; \
    fi
if test -z "$ZTST_handler"; then \
      ZTST_handler=runtests.zsh; \
    fi; \
    if ZTST_testlist="`for f in ./Y01*.ztst; \
           do echo $f; done`" \
     ZTST_srcdir="." \
     ZTST_exe=../Src/zsh \
     ../Src/zsh +Z -f ./$ZTST_handler; then \
     stat=0; \
    else \
     stat=1; \
    fi; \
    sleep 1; \
    rm -rf Modules .zcompdump; \
    exit $stat
./Y01completion.ztst: starting.
./Y01completion.ztst: all tests successful.
**************************************
1 successful test script, 0 failures, 0 skipped
**************************************

Re: Y02 test failure

[Lawrence Velázquez]

On Fri, Nov 12, 2021, at 4:49 PM, Bart Schaefer wrote:
> zsh-5.8-542-gc7f4634
>
>   524  make realclean
>   526  ./Util/preconfig
>   527  ./configure
>   528  make
>   529  make check TESTNUM=Y01

This passes for me on macOS as well, in a clean tree.

-- 
vq

Re: Y02 test failure

[Marlon Richert]

On Sat, Nov 13, 2021 at 12:01 AM Lawrence Velázquez <larryv@zsh.org> wrote:
>
> On Fri, Nov 12, 2021, at 4:49 PM, Bart Schaefer wrote:
> > zsh-5.8-542-gc7f4634
> >
> >   524  make realclean
> >   526  ./Util/preconfig
> >   527  ./configure
> >   528  make
> >   529  make check TESTNUM=Y01
>
> This passes for me on macOS as well, in a clean tree.

It now passes for me, too. No idea what changed. I wonder if something
in my shell environment could have affected it?

Re: Y01 test failure

[Marlon Richert]

On Sat, Nov 13, 2021 at 7:15 PM Marlon Richert <marlon.richert@gmail.com> wrote:
>
> On Sat, Nov 13, 2021 at 12:01 AM Lawrence Velázquez <larryv@zsh.org> wrote:
> >
> > On Fri, Nov 12, 2021, at 4:49 PM, Bart Schaefer wrote:
> > > zsh-5.8-542-gc7f4634
> > >
> > >   524  make realclean
> > >   526  ./Util/preconfig
> > >   527  ./configure
> > >   528  make
> > >   529  make check TESTNUM=Y01
> >
> > This passes for me on macOS as well, in a clean tree.
>
> It now passes for me, too. No idea what changed. I wonder if something
> in my shell environment could have affected it?

And now it fails for me again in the same way, but this time on Ubuntu:

% make TESTNUM=Y01 check
cd Test ; make check
make[1]: Entering directory '/home/marlon/Git/zsh/Test'
if test -n "gcc"; then \
  cd .. && DESTDIR= \
  make MODDIR=`pwd`/Test/Modules install.modules > /dev/null; \
fi
if test -z "$ZTST_handler"; then \
  ZTST_handler=runtests.zsh; \
fi; \
if ZTST_testlist="`for f in ./Y01*.ztst; \
           do echo $f; done`" \
 ZTST_srcdir="." \
 ZTST_exe=../Src/zsh \
 ../Src/zsh +Z -f ./$ZTST_handler; then \
 stat=0; \
else \
 stat=1; \
fi; \
sleep 1; \
rm -rf Modules .zcompdump; \
exit $stat
./Y01completion.ztst: starting.
--- /tmp/zsh.ztst.169509/ztst.out       2021-11-29 23:19:58.088524580 +0200
+++ /tmp/zsh.ztst.169509/ztst.tout      2021-11-29 23:19:58.120524372 +0200
@@ -1 +1 @@
-line: {ztst.zsh }{}
+line: {ztst.zsh}{}
Test ./Y01completion.ztst failed: output differs from expected as
shown above for:
  comptesteval "path=( $ZTST_srcdir:A )"
  comptest $'zt\t'
Was testing: command
./Y01completion.ztst: test failed.
**************************************
0 successful test scripts, 1 failure, 0 skipped
**************************************
make[1]: *** [Makefile:190: check] Error 1
make[1]: Leaving directory '/home/marlon/Git/zsh/Test'
make: *** [Makefile:263: check] Error 2

What could be causing this?

This happens even when I run this in a new terminal that is configured
to start with /bin/sh. So, I don't think it's anything in my dotfiles
that's causing this.

Re: Y01 test failure

[Jun. T]

> 2021/11/30 6:43, Marlon Richert <marlon.richert@gmail.com> wrote:
> 
> -line: {ztst.zsh }{}
> +line: {ztst.zsh}{}

Don't you have a file like 'ztst.zsh.new' or 'ztst.zsh.v1' or such
in the Test/ directory?

Re: Y01 test failure

[Bart Schaefer]

On Tue, Nov 30, 2021 at 6:58 AM Jun. T <takimoto-j@kba.biglobe.ne.jp> wrote:
>
> > 2021/11/30 6:43, Marlon Richert <marlon.richert@gmail.com> wrote:
> >
> > -line: {ztst.zsh }{}
> > +line: {ztst.zsh}{}
>
> Don't you have a file like 'ztst.zsh.new' or 'ztst.zsh.v1' or such
> in the Test/ directory?

That's got to be it.  For example, I can reproduce this test failure
if there is an emacs backup file Test/ztst.zsh~ present.