Multiple as-installed test suite failures under different circumstances with 5.1-test-1

Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Axel Beckert]

Hi,

while zsh's test suite seems to pass at build time, when running zsh's
tests on the installed zsh (package) as done in Debian now for quite a
while (since 5.0.6, see
http://ci.debian.net/packages/z/zsh/unstable/amd64/), the test suite fails at two
different places dependening on where I run test suite:

In our Jenkins CI, at least three failures are shown on
http://jenkins.grml.org/view/Debian/job/zsh-binaries/architecture=amd64/209/console:

/tmp/adttmp.mhKVzv/Test/A02alias.ztst: starting.
This test hangs the shell when it fails...
*** /tmp/zsh.ztst.out.546 Tue Sep  8 21:07:04 2015
--- /tmp/zsh.ztst.tout.546    Tue Sep  8 21:07:04 2015
***************
*** 1,2 ****
  foo is a suffix alias for print
! foo: suffix alias
--- 1,2 ----
  foo is a suffix alias for print
! foo: alias
Test /tmp/adttmp.mhKVzv/Test/A02alias.ztst failed: output differs from expected as shown above for:
  alias -s foo=print
  type bar.foo; type -w bar.foo
  unalias -as
Was testing: unalias -as
/tmp/adttmp.mhKVzv/Test/A02alias.ztst: test failed.

/tmp/adttmp.mhKVzv/Test/A04redirect.ztst: starting.
*** /tmp/zsh.ztst.out.646		  Tue Sep  8 21:07:05 2015
--- /tmp/zsh.ztst.tout.646		  Tue Sep  8 21:07:05 2015
***************
*** 1,8 ****
  fn-two-heres () {
    cat <<x <<y
  foo
! x
! bar
  y
  }
  foo
--- 1,7 ----
  fn-two-heres () {
    cat <<x <<y
  foo
! xbar
  y
  }
  foo
***************
*** 11,16 ****
  bar
	cat <<x <<y
  foo
! x
! bar
  y
--- 10,14 ----
  bar
	cat <<x <<y
  foo
! xbar
  y
Test /tmp/adttmp.mhKVzv/Test/A04redirect.ztst failed: output differs from expected as shown above for:
  fn-two-heres() {
    cat <<-x <<-y
    foo
    x
    bar
    y
  }
  which -x2 fn-two-heres
  fn-two-heres
  eval "$(which -x2 fn-two-heres)"
  fn-two-heres
  print $functions[fn-two-heres]
Error output:
(eval):6: parse error near `\n'
Was testing: Two here-documents in a line are shown correctly.
/tmp/adttmp.mhKVzv/Test/A04redirect.ztst: test failed.

/tmp/adttmp.mhKVzv/Test/A05execution.ztst: starting.
Unable to change MONITOR option
This test takes 5 seconds to fail...
Skipping pipe leak test, requires MONITOR option
This test takes 3 seconds and hangs the shell when it fails...
[and then it hangs]

Running the test suite against my locally built and installed Debian
package of zsh 5.1-test-1, it passes all the tests above, but fails
later as follows:

/tmp/adttmp.ga8JYP/Test/V06parameter.ztst: starting.
*** /tmp/zsh.ztst.out.22805     Tue Sep  8 23:25:46 2015
--- /tmp/zsh.ztst.tout.22805    Tue Sep  8 23:25:46 2015
***************
*** 1,10 ****
  Started functrace.zsh
! 3 + +
  Inside function fn
! 2 + ./functrace.zsh:10 + ./functrace.zsh:5
  Inside autofn
! 2 + ./functrace.zsh:20 + ./autofn:0
  Inside autofn
! 2 + ./functrace.zsh:21 + ./autofn:0
  In sourced file
! 2 + ./functrace.zsh:22 + ./sourcedfile:0
--- 1,10 ----
  Started functrace.zsh
! 3 + zsh/parameter + zsh/parameter
  Inside function fn
! 2 + zsh/parameter + zsh/parameter
  Inside autofn
! 2 + zsh/parameter + zsh/parameter
  Inside autofn
! 2 + zsh/parameter + zsh/parameter
  In sourced file
! 2 + zsh/parameter + zsh/parameter
Test /tmp/adttmp.ga8JYP/Test/V06parameter.ztst failed: output differs from expected as shown above for:
  print 'print In sourced file
  print $LINENO + $functrace + $funcsourcetrace
  ' >sourcedfile
  print -r -- 'print Started functrace.zsh
  module_path=(./Modules)
  print $LINENO + $functrace + $funcsourcetrace
  :
  fn() {
    print Inside function $0
    print $LINENO + $functrace + $funcsourcetrace
  }
  :
  fn
  :
  fpath=(. $fpath)
  :
  echo '\''print Inside $0
    print $LINENO + $functrace + $funcsourcetrace
  '\'' >autofn
  :
  autoload autofn
  :
  autofn
  autofn
  . ./sourcedfile' >functrace.zsh
  $ZTST_testdir/../Src/zsh +Z -f ./functrace.zsh
Error output:
./functrace.zsh:3: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
./functrace.zsh:3: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
fn:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
fn:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
autofn:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
autofn:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
autofn:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
autofn:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
./sourcedfile:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
./sourcedfile:2: failed to load module `zsh/parameter': ./Modules/zsh/parameter.so: cannot open shared object file: No such file or directory
Was testing: Function tracing
/tmp/adttmp.ga8JYP/Test/V06parameter.ztst: test failed.

(It is possible that not all necessary files have been copied from the
zsh source directory to the directory from which the tests are run if
there are additional files necessary since 5.1 -- which passed these
tests at least on our Jenkins CI.)

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 517 bytes --]

On Sep 8, 2015 5:37 PM, "Axel Beckert" <abe@deuxchevaux.org> wrote:
>
> Hi,
>
> while zsh's test suite seems to pass at build time, when running zsh's
> tests on the installed zsh (package) as done in Debian now for quite a
> while (since 5.0.6, see
> http://ci.debian.net/packages/z/zsh/unstable/amd64/), the test suite
fails at two
> different places dependening on where I run test suite:

This looks to me as if the wrong zsh binary is being executed for some of
the tests.  Are there two zsh packages installed?

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Peter Stephenson]

On Tue, 8 Sep 2015 18:05:14 -0500
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sep 8, 2015 5:37 PM, "Axel Beckert" <abe@deuxchevaux.org> wrote:
> > while zsh's test suite seems to pass at build time, when running zsh's
> > tests on the installed zsh (package) as done in Debian now for quite a
> > while (since 5.0.6, see
> > http://ci.debian.net/packages/z/zsh/unstable/amd64/), the test suite
> fails at two
> > different places dependening on where I run test suite:
> 
> This looks to me as if the wrong zsh binary is being executed for some of
> the tests.  Are there two zsh packages installed?

Yep, those tests are new, to cover recent changes to the shell.
I'm 99% convinced you're just testing an older version, else it
doesn't really make sense.

pws

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Axel Beckert]

Hi,

On Wed, Sep 09, 2015 at 09:27:58AM +0100, Peter Stephenson wrote:
> On Tue, 8 Sep 2015 18:05:14 -0500
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > On Sep 8, 2015 5:37 PM, "Axel Beckert" <abe@deuxchevaux.org> wrote:
> > > while zsh's test suite seems to pass at build time, when running zsh's
> > > tests on the installed zsh (package) as done in Debian now for quite a
> > > while (since 5.0.6, see
> > > http://ci.debian.net/packages/z/zsh/unstable/amd64/), the test suite
> > fails at two
> > > different places dependening on where I run test suite:
> > 
> > This looks to me as if the wrong zsh binary is being executed for some of
> > the tests.  Are there two zsh packages installed?
> 
> Yep, those tests are new, to cover recent changes to the shell.
> I'm 99% convinced you're just testing an older version, else it
> doesn't really make sense.

Thanks for that hint. Will check the remainder of the log if e.g. an
automated installation step failed and has been unrecognised.

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Axel Beckert]

Hi,

On Wed, Sep 09, 2015 at 01:04:04PM +0200, Axel Beckert wrote:
> > > This looks to me as if the wrong zsh binary is being executed for some of
> > > the tests.  Are there two zsh packages installed?
> > 
> > Yep, those tests are new, to cover recent changes to the shell.
> > I'm 99% convinced you're just testing an older version, else it
> > doesn't really make sense.
> 
> Thanks for that hint. Will check the remainder of the log if e.g. an
> automated installation step failed and has been unrecognised.

Indeed, in the Jenkins job, at least in all cases in the past two
days, 5.1 was installed instead of 5.1-test-1. We're still
investigating why this happens (and why it didn't happen in the past).

We though had a test suite failure at _build_ time (i.e. "make test"
directly after the build) once and only once withing 5 build runs so
far, so it may be a kind of race condition:

Running test: SH_NULLCMD option
*** /tmp/zsh.ztst.err.3773	Wed Sep  9 13:06:16 2015
--- /tmp/zsh.ztst.terr.3773	Wed Sep  9 13:06:16 2015
***************
*** 0 ****
--- 1,2 ----
+ fn:echo:2: write error: broken pipe
+ fn:2: write error: inappropriate ioctl for device
Test ../../Test/E01options.ztst failed: error output differs from expected as shown above for:
  print this is bar >bar
  fn() {
    local NULLCMD=cat READNULLCMD=cat
    echo hello | >foo
    cat foo
    <bar
  }
  setopt shnullcmd
  print option set
  fn
  unsetopt shnullcmd
  print option unset
  fn
  rm -f foo bar
Was testing: SH_NULLCMD option
../../Test/E01options.ztst: test failed.
../../Test/E02xtrace.ztst: starting.

I wonder if that's another case of a missing "2>/dev/null" like we
fixed a few before 5.1?

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Peter Stephenson]

On Wed, 9 Sep 2015 16:10:27 +0200
Axel Beckert <abe@deuxchevaux.org> wrote:
> We though had a test suite failure at _build_ time (i.e. "make test"
> directly after the build) once and only once withing 5 build runs so
> far, so it may be a kind of race condition:
> 
>...
> Test ../../Test/E01options.ztst failed: error output differs from expected as shown above for:
>...
> Was testing: SH_NULLCMD option
>...
> ../../Test/E01options.ztst: test failed.
> ../../Test/E02xtrace.ztst: starting.
> + fn:echo:2: write error: broken pipe
> + fn:2: write error: inappropriate ioctl for device
>...
> I wonder if that's another case of a missing "2>/dev/null" like we
> fixed a few before 5.1?

OK, I suppose

  echo hello | >foo

in the case of SH_NULLCMD, which is equivalent to

  echo hello | : >foo

is indeed liable to the race in question, which is entirely benign for
this test.  It shouldn't fail for the case where this is doing a cat,
but the stdout test output already checks we picked that up, so
suppressing the error looks reasonable.

How about changing it to

  { echo hello | >foo } 2>/dev/null

?  Is that good enough?

pws

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Axel Beckert]

Hi Peter,

On Wed, Sep 09, 2015 at 03:39:02PM +0100, Peter Stephenson wrote:
> > ../../Test/E01options.ztst: test failed.
> > + fn:echo:2: write error: broken pipe
> > + fn:2: write error: inappropriate ioctl for device
> >...
> > I wonder if that's another case of a missing "2>/dev/null" like we
> > fixed a few before 5.1?
> 
> OK, I suppose
> 
>   echo hello | >foo
> 
> in the case of SH_NULLCMD, which is equivalent to
> 
>   echo hello | : >foo
> 
> is indeed liable to the race in question, which is entirely benign for
> this test.  It shouldn't fail for the case where this is doing a cat,
> but the stdout test output already checks we picked that up, so
> suppressing the error looks reasonable.
>
> How about changing it to
> 
>   { echo hello | >foo } 2>/dev/null
> 
> ?  Is that good enough?

Looks fine and the two builds I did since then worked fine, too. Then
again, I can't really say since we saw this only once so far and
couldn't reproduce since then.

I'm keeping this now in the Debian package as a patch for while unless
you think it's good enough to be committed upstream anyways.

		Kind regards, Axel
-- 
/~\  Plain Text Ribbon Campaign                   | Axel Beckert
\ /  Say No to HTML in E-Mail and News            | abe@deuxchevaux.org  (Mail)
 X   See http://www.nonhtmlmail.org/campaign.html | abe@noone.org (Mail+Jabber)
/ \  I love long mails: http://email.is-not-s.ms/ | http://abe.noone.org/ (Web)

Re: Multiple as-installed test suite failures under different circumstances with 5.1-test-1

[Peter Stephenson]

On Thu, 10 Sep 2015 09:31:01 +0200
Axel Beckert <abe@deuxchevaux.org> wrote:
> I'm keeping this now in the Debian package as a patch for while unless
> you think it's good enough to be committed upstream anyways.

I've committed it: I can't see a problem with it.

pws