D03 test hang on cygwin with latest sources

D03 test hang on cygwin with latest sources

[Vin Shelton]

Peter -

It looks like something bad happened with your process redirection
changes.  After building from the latest CVS sources, I ran the
testsuite and got a hang in D03.  A ZTST_verbose=2 trace follows.

ZTST_verbose=2 make TESTNUM=D03
if test -n "/usr/bin/gcc-4"; then \
          cd .. && DESTDIR= \
          make MODDIR=`pwd`/Test/Modules install.modules > /dev/null; \
        fi
if ZTST_testlist="`for f in /usr/local/src/zsh-2008-11-20/Test/D03*.ztst; \
           do echo $f; done`" \
         ZTST_srcdir="/usr/local/src/zsh-2008-11-20/Test" \
         ZTST_exe=../Src/zsh.exe \
         ../Src/zsh.exe +Z -f
/usr/local/src/zsh-2008-11-20/Test/runtests.zsh; then \
         stat=0; \
        else \
         stat=1; \
        fi; \
        rm -rf Modules .zcompdump; \
        exit $stat
/usr/local/src/zsh-2008-11-20/Test/D03procsubst.ztst: starting.
ZTST_getsect: read section name: prep
ZTST_getchunk: read code chunk:
  if grep '#define PATH_DEV_FD' $ZTST_testdir/../config.h > /dev/null 2>&1 ||
     grep '#define HAVE_FIFOS' $ZTST_testdir/../config.h > /dev/null 2>&1; then
    mkdir procsubst.tmp
    cd procsubst.tmp
    print 'First\tSecond\tThird\tFourth' >FILE1
    print 'Erste\tZweite\tDritte\tVierte' >FILE2
  else
    ZTST_unimplemented="process substitution is not supported"
    true
  fi
ZTST_execchunk: status 0
ZTST_getchunk: read code chunk:
  function copycat { cat "$@" }
ZTST_execchunk: status 0
ZTST_getchunk: read code chunk:

ZTST_getsect: read section name: test
ZTST_test: looking for new test
ZTST_test: examining line:
  paste <(cut -f1 FILE1) <(cut -f3 FILE2)
ZTST_getchunk: read code chunk:
  paste <(cut -f1 FILE1) <(cut -f3 FILE2)
ZTST_test: examining line:
>First  Dritte
ZTST_getredir: read redir for '>':
First   Dritte
ZTST_test: examining line:

Running test: <(...) substitution
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.1164, output: /tmp/zsh.ztst.out.1164, error:
/tmp/zsh.ztst.terr.1164
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
First   Dritte
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  { paste <(cut -f2 FILE1) <(cut -f4 FILE2) } > >(sed 's/e/E/g' >OUTFILE)
ZTST_getchunk: read code chunk:
  { paste <(cut -f2 FILE1) <(cut -f4 FILE2) } > >(sed 's/e/E/g' >OUTFILE)
  cat OUTFILE
ZTST_test: examining line:
>SEcond ViErtE
ZTST_getredir: read redir for '>':
SEcond  ViErtE
ZTST_test: examining line:

Running test: >(...) substitution
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.1164, output: /tmp/zsh.ztst.out.1164, error:
/tmp/zsh.ztst.terr.1164
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
SEcond  ViErtE
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  diff =(cat FILE1) =(cat FILE2)
ZTST_getchunk: read code chunk:
  diff =(cat FILE1) =(cat FILE2)
ZTST_test: examining line:
>1c1
ZTST_getredir: read redir for '>':
1c1
< First Second  Third   Fourth
---
> Erste Zweite  Dritte  Vierte
ZTST_test: examining line:

Running test: =(...) substituion
ZTST_test: expecting status: 1
Input: /tmp/zsh.ztst.in.1164, output: /tmp/zsh.ztst.out.1164, error:
/tmp/zsh.ztst.terr.1164
ZTST_execchunk: status 1
ZTST_test: test produced standard output:
1c1
< First Second  Third   Fourth
---
> Erste Zweite  Dritte  Vierte
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  copycat <(print First) <(print Zweite)
ZTST_getchunk: read code chunk:
  copycat <(print First) <(print Zweite)
ZTST_test: examining line:
>First
ZTST_getredir: read redir for '>':
First
Zweite
ZTST_test: examining line:

Running test: FDs remain open for external commands called from functions
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.1164, output: /tmp/zsh.ztst.out.1164, error:
/tmp/zsh.ztst.terr.1164
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
First
Zweite
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  catfield2() {
ZTST_getchunk: read code chunk:
  catfield2() {
    local -a args
    args=(${(s.,.)1})
    print $args[1]
    cat $args[2]
    print $args[3]
  }
  catfield2 up,<(print $'\x64'own),sideways
ZTST_test: examining line:
>up
ZTST_getredir: read redir for '>':
up
down
sideways
ZTST_test: examining line:

Running test: <(...) when embedded within an argument
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.1164, output: /tmp/zsh.ztst.out.1164, error:
/tmp/zsh.ztst.terr.1164
ZTST_execchunk: status 0
ZTST_test: test produced standard output:
up
down
sideways
ZTST_test: and standard error:

Test successful.
ZTST_test: looking for new test
ZTST_test: examining line:

ZTST_test: examining line:
  outputfield2() {
ZTST_getchunk: read code chunk:
  outputfield2() {
    local -a args
    args=(${(s.,.)1})
    print $args[1]
    echo 'How sweet the moonlight sits upon the bank' >$args[2]
    print $args[3]
  }
  outputfield2 muddy,>(sed -e s/s/th/g >outputfield2.txt),vesture
  # yuk
  while [[ ! -e outputfield2.txt || ! -s outputfield2.txt ]]; do :; done
  cat outputfield2.txt
ZTST_test: examining line:
>muddy
ZTST_getredir: read redir for '>':
muddy
vesture
How thweet the moonlight thitth upon the bank
ZTST_test: examining line:

Running test: >(...) when embedded within an argument
ZTST_test: expecting status: 0
Input: /tmp/zsh.ztst.in.1164, output: /tmp/zsh.ztst.out.1164, error:
/tmp/zsh.ztst.terr.1164
make: *** [check] Interrupt


The last time I built the shell on this platform, 2008-10-12, this test passed.

Please let me know if you need any further diagnosis.

Regards,
  Vin

Re: D03 test hang on cygwin with latest sources

[Bart Schaefer]

On Nov 20, 11:08pm, Vin Shelton wrote:
}
} Peter -
} 
} It looks like something bad happened with your process redirection
} changes.  After building from the latest CVS sources, I ran the
} testsuite and got a hang in D03.  A ZTST_verbose=2 trace follows.
} 
} The last time I built the shell on this platform, 2008-10-12, this
} test passed.

I'm not sure historical success means anything here, because there
previously was no test of >(...) except in the context of > >(...).

The pre-existing test for > >(...) has the somewhat cryptic comment

# slightly desperate hack to force >(...) to be synchronous

I note there's no similar synchronicity hack in the new test, and the
debug output is not sufficient to determine whether the shell is in
fact hung, or whether it's in an infinite loop running

  while [[ ! -e outputfield2.txt || ! -s outputfield2.txt ]]; do :; done

Re: D03 test hang on cygwin with latest sources

[Peter Stephenson]

On Thu, 20 Nov 2008 21:12:44 -0800
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Nov 20, 11:08pm, Vin Shelton wrote:
> }
> } Peter -
> } 
> } It looks like something bad happened with your process redirection
> } changes.  After building from the latest CVS sources, I ran the
> } testsuite and got a hang in D03.  A ZTST_verbose=2 trace follows.
> } 
> } The last time I built the shell on this platform, 2008-10-12, this
> } test passed.
> 
> I'm not sure historical success means anything here, because there
> previously was no test of >(...) except in the context of > >(...).

Right, the behaviour isn't new.  It seems as if the subprocess is
reading EOF from the input.

% foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! }
% foo >(sleep 1; read foo || print Failed)
/proc/self/fd/12
Failed

I don't understand why it would do that, given the order in which things
apparently happen (there's roughly a second gap between the messages, as
you'd expect).  Waiting the other way round, I get

% foo() { print $1; (sleep 1; print hello) >$1; [[ -e $1 ]] || print Ouch! }
% foo >(read foo || print Failed)
/proc/self/fd/12
foo: write error: bad file descriptor
Failed

I printed $ERRNO and it's zero.  Note that the read in the subprocess only
fails *after* the attempt to write fails.

It's also not clear to me why this is so different from "print hello | read
foo", which works, except to note that pipes in this case exchange a sync
word.

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D03 test hang on cygwin with latest sources

[Phil Pennock]

On 2008-11-24 at 17:32 +0000, Peter Stephenson wrote:
> Right, the behaviour isn't new.  It seems as if the subprocess is
> reading EOF from the input.
> 
> % foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! }
> % foo >(sleep 1; read foo || print Failed)
> /proc/self/fd/12
> Failed

...% zsh -f
redoubt% echo $ZSH_VERSION 
4.3.6
redoubt% foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! } 
redoubt% foo >(sleep 1; read foo || print Failed)
/tmp/zshQCDdM0
redoubt%

Looks like a regression after all.
-Phil

Re: D03 test hang on cygwin with latest sources

[Peter Stephenson]

Phil Pennock wrote:
> On 2008-11-24 at 17:32 +0000, Peter Stephenson wrote:
> > Right, the behaviour isn't new.  It seems as if the subprocess is
> > reading EOF from the input.
> > 
> > % foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! }
> > % foo >(sleep 1; read foo || print Failed)
> > /proc/self/fd/12
> > Failed
> 
> ...% zsh -f
> redoubt% echo $ZSH_VERSION 
> 4.3.6
> redoubt% foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! } 
> redoubt% foo >(sleep 1; read foo || print Failed)
> /tmp/zshQCDdM0
> redoubt%
> 
> Looks like a regression after all.

Did you try it in 4.3.9?  I tried it on three different versions, before
and after 4.3.6, and got the same result.  If your system is doing
something different I'd like to find out what.

I *can* get the test to pass if I undefine PATH_DEV_FD and define
HAVE_FIFOS, so it looks like this is the way forward.  Some
investigation of why HAVE_FIFOS doesn't get define automatically would
be useful.

(I tried adding a "sync" pipe to getpipe() in the same way as in the
other pipe code, but it didn't seem to help.)

I'm also finding that I can read files with mode 000, so the [[ -r
... ]] test fails (but it's clearly doing the right thing, since "cat"
on the file works, too).  This may be because I have administrator
rights.  Unless some Cygwin expert can do this better, I'll apply the
following.

This is with Cygwin updated from the stable versions yesterday.

Index: Test/C02cond.ztst
===================================================================
RCS file: /cvsroot/zsh/zsh/Test/C02cond.ztst,v
retrieving revision 1.22
diff -u -r1.22 C02cond.ztst
--- Test/C02cond.ztst	26 Feb 2008 20:50:13 -0000	1.22
+++ Test/C02cond.ztst	25 Nov 2008 09:54:00 -0000
@@ -94,6 +94,10 @@
   if (( EUID == 0 )); then
     print -u$ZTST_fd 'Warning: Not testing [[ ! -r file ]] (root reads anything)'
     [[ -r zerolength && -r unmodish ]]
+  elif [[ $OSTYPE = cygwin ]]; then
+    print -u$ZTST_fd 'Warning: Not testing [[ ! -r file ]]
+   (all files created by user may be readable)'
+   [[ -r zerolength ]]
   else
     [[ -r zerolength && ! -r unmodish ]]
   fi


-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D03 test hang on cygwin with latest sources

[Peter A. Castro]

On Tue, 25 Nov 2008, Peter Stephenson wrote:

> Phil Pennock wrote:
>> On 2008-11-24 at 17:32 +0000, Peter Stephenson wrote:
>>> Right, the behaviour isn't new.  It seems as if the subprocess is
>>> reading EOF from the input.
>>>
>>> % foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! }
>>> % foo >(sleep 1; read foo || print Failed)
>>> /proc/self/fd/12
>>> Failed
>>
>> ...% zsh -f
>> redoubt% echo $ZSH_VERSION
>> 4.3.6
>> redoubt% foo() { print $1; print hello >$1; [[ -e $1 ]] || print Ouch! }
>> redoubt% foo >(sleep 1; read foo || print Failed)
>> /tmp/zshQCDdM0
>> redoubt%
>>
>> Looks like a regression after all.
>
> Did you try it in 4.3.9?  I tried it on three different versions, before
> and after 4.3.6, and got the same result.  If your system is doing
> something different I'd like to find out what.
>
> I *can* get the test to pass if I undefine PATH_DEV_FD and define
> HAVE_FIFOS, so it looks like this is the way forward.  Some
> investigation of why HAVE_FIFOS doesn't get define automatically would
> be useful.

Well, configure.ac has an explicit check for cygwin and disables the feature:

dnl -----------
dnl named FIFOs
dnl -----------
AC_CACHE_CHECK(if named FIFOs work,
zsh_cv_sys_fifo,
[if test "$host_os" = cygwin; then
zsh_cv_sys_fifo=no
else


The other thing about this is that the fifo test code fails but the
reason appears to be a race condition between the parent and child
creating the fifo.  I've modified the example thus:

#include <fcntl.h>
#include <signal.h>
main()
{
     char c;
     int fd;
     int pid, ret;
     unlink("/tmp/fifo$$");
#ifdef HAVE_MKFIFO
     if(mkfifo("/tmp/fifo$$", 0600) < 0)
#else
     if(mknod("/tmp/fifo$$", 0010600, 0) < 0)
#endif
 	exit(1);
     pid = fork();
     if(pid < 0)
 	exit(1);
     if(pid) {
 	fd = open("/tmp/fifo$$", O_RDONLY);
 	exit(fd < 0 || read(fd, &c, 1) != 1 || c != 'x');
     }
     sleep(1);  /* prevent race */
     fd = open("/tmp/fifo$$", O_WRONLY);
     ret = (fd < 0 || write(fd, "x", 1) < 1);
     unlink("/tmp/fifo$$");
     exit(ret);
}


With this, the test runs, and HAVE_FIFOS gets defined.  I hacked
configure to not set PATH_DEV_FD too.  :-)

With the above changes the 'foo' test above works, though
Test/C02cond.ztst still generates an error.

Cygwin (really Windows) has some issues with fifos and the order in which
they are created, and for the longest time fifos were really just not
usable under Cygwin.  Recent versions of Cygwin have updated the
fifo/pipe code but it's still subject to Windows underlying functions.

I understand the up coming Cygwin-1.7 will have a completely new
implemention of pipes which should behave more like what the POSIX spec
says it should.

I could probably provide a sample configure.ac for all of this if you'd
like, but the changes are pretty trivial.


> (I tried adding a "sync" pipe to getpipe() in the same way as in the
> other pipe code, but it didn't seem to help.)
>
> I'm also finding that I can read files with mode 000, so the [[ -r
> ... ]] test fails (but it's clearly doing the right thing, since "cat"
> on the file works, too).  This may be because I have administrator
> rights.  Unless some Cygwin expert can do this better, I'll apply the
> following.
>
> This is with Cygwin updated from the stable versions yesterday.
>
> Index: Test/C02cond.ztst
> ===================================================================
> RCS file: /cvsroot/zsh/zsh/Test/C02cond.ztst,v
> retrieving revision 1.22
> diff -u -r1.22 C02cond.ztst
> --- Test/C02cond.ztst	26 Feb 2008 20:50:13 -0000	1.22
> +++ Test/C02cond.ztst	25 Nov 2008 09:54:00 -0000
> @@ -94,6 +94,10 @@
>   if (( EUID == 0 )); then
>     print -u$ZTST_fd 'Warning: Not testing [[ ! -r file ]] (root reads anything)'
>     [[ -r zerolength && -r unmodish ]]
> +  elif [[ $OSTYPE = cygwin ]]; then
> +    print -u$ZTST_fd 'Warning: Not testing [[ ! -r file ]]
> +   (all files created by user may be readable)'
> +   [[ -r zerolength ]]
>   else
>     [[ -r zerolength && ! -r unmodish ]]
>   fi

-- 
Peter A. Castro <doctor@fruitbat.org> or <Peter.Castro@oracle.com>
 	"Cats are just autistic Dogs" -- Dr. Tony Attwood

Re: D03 test hang on cygwin with latest sources

[Peter Stephenson]

On Tue, 25 Nov 2008 15:26:09 -0800 (PST)
"Peter A. Castro" <doctor@fruitbat.org> wrote:
> Well, configure.ac has an explicit check for cygwin and disables the feature:
> 
> dnl -----------
> dnl named FIFOs
> dnl -----------
> AC_CACHE_CHECK(if named FIFOs work,
> zsh_cv_sys_fifo,
> [if test "$host_os" = cygwin; then
> zsh_cv_sys_fifo=no
> else

So that didn't take long to check, after all...

> The other thing about this is that the fifo test code fails but the
> reason appears to be a race condition between the parent and child
> creating the fifo.  I've modified the example thus:

With most FIFO implementations this wouldn't be necessary since after the
FIFO is created both the read and the write would block if there's nothing
at the other end.  I therefore don't know enough to provide a robust fix.
I've simply changed the Cygwin-specific code above.

> With the above changes the 'foo' test above works, though
> Test/C02cond.ztst still generates an error.

I fixed that, so all tests pass (except the multibyte test which is
skipped).

Thanks for the notes.

By the way, do you know what the status is under Vista?  I'm still getting
more than occasional fatal DLL conflicts.  Last time I checked the only
fixes for this were ad-hoc magic.

Index: configure.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/configure.ac,v
retrieving revision 1.113
diff -u -r1.113 configure.ac
--- configure.ac	18 Nov 2008 10:14:35 -0000	1.113
+++ configure.ac	1 Dec 2008 12:12:22 -0000
@@ -1787,9 +1787,17 @@
 AH_TEMPLATE([PATH_DEV_FD],
 [Define to the path of the /dev/fd filesystem.])
 AC_CACHE_CHECK(for /dev/fd filesystem, zsh_cv_sys_path_dev_fd,
+[if test "$host_os" = cygwin; then
+dnl In current (2008/12/01) versions of Cygwin these are present but don't
+dnl seem to work smoothly for process substitution; no great surprise
+dnl since getting processes to work at all on Cygwin is a big challenge.
+dnl We'll rely on FIFOs, since they do what we need.
+zsh_cv_sys_path_dev_fd=no
+else
 [for zsh_cv_sys_path_dev_fd in /proc/self/fd /dev/fd no; do
    test x`echo ok|(exec 3<&0; cat $zsh_cv_sys_path_dev_fd/3 2>/dev/null;)` = xok && break
- done])
+ done]
+fi])
 if test x$zsh_cv_sys_path_dev_fd != xno; then
   AC_DEFINE_UNQUOTED(PATH_DEV_FD, "$zsh_cv_sys_path_dev_fd")
 fi
@@ -1954,10 +1962,13 @@
 dnl -----------
 dnl named FIFOs
 dnl -----------
+dnl
+dnl Named FIFOs work well enough on recent versions of Cygwin
+dnl to provide what we want.  Simply enable them.
 AC_CACHE_CHECK(if named FIFOs work,
 zsh_cv_sys_fifo,
 [if test "$host_os" = cygwin; then
-zsh_cv_sys_fifo=no
+zsh_cv_sys_fifo=yes
 else
 AC_TRY_RUN([
 #include <fcntl.h>

-- 
Peter Stephenson <pws@csr.com>                  Software Engineer
CSR PLC, Churchill House, Cambridge Business Park, Cowley Road
Cambridge, CB4 0WZ, UK                          Tel: +44 (0)1223 692070

Re: D03 test hang on cygwin with latest sources

[Peter A. Castro]

On Mon, 1 Dec 2008, Peter Stephenson wrote:

> On Tue, 25 Nov 2008 15:26:09 -0800 (PST)
> "Peter A. Castro" <doctor@fruitbat.org> wrote:
[snip]
> Thanks for the notes.
>
> By the way, do you know what the status is under Vista?  I'm still getting
> more than occasional fatal DLL conflicts.  Last time I checked the only
> fixes for this were ad-hoc magic.

Getting Cygwin to work at all on Vista is still a work in progress.
There are many, many problems, none of which are truely understood.
It's mostly a terrible mix of UACs, Admin auth and cat hair.
Ad-hoc magic is as good as anything else for the moment.  Sorry.

-- 
Peter A. Castro <doctor@fruitbat.org> or <Peter.Castro@oracle.com>
 	"Cats are just autistic Dogs" -- Dr. Tony Attwood